{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8f224944",
   "metadata": {},
   "source": [
    "# 实现可以产生文本的GPT大语言模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1de34e67",
   "metadata": {},
   "source": [
    "在本系列视频中，我们将实现类GPT的大语言模型，下个系列视频将介绍如何训练这个模型。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62db310d",
   "metadata": {},
   "source": [
    "![](./images/framework.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d1c4dc3",
   "metadata": {},
   "source": [
    "## 大语言模型架构"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac5f604d",
   "metadata": {},
   "source": [
    "- 环境搭建部分讨论了GPT、LLAMA等模型，它们基于原始Transformer架构的解码器部分，采用逐词生成方式；\n",
    "- 因此这类大语言模型通常被称为\"类解码器\"大语言模型（decoder-like LLMs）；\n",
    "- 与传统深度学习模型相比，大语言模型的规模更大，这主要体现于海量的参数规模而非代码量；\n",
    "- 我们将看到大语言模型架构中存在大量重复的模块组件。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a78b919f",
   "metadata": {},
   "source": [
    "![](./images/modules.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbacc8ff",
   "metadata": {},
   "source": [
    "- 在前几章中，为便于示意图展示，我们使用了较小的词元输入输出嵌入维度，确保整体结构可呈现；\n",
    "- 本系列视频将采用接近GPT-2小规模模型的嵌入维度与模型尺寸；\n",
    "- 我们将具体编码实现最小规模的GPT-2架构（1.24亿参数），其设计遵循Radford等人论文[Language Models are Unsupervised Multitask Learners](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)（注：初始报告记录为1.17亿参数，后在模型权重库中修正为1.24亿）；\n",
    "- 后面将演示如何将预训练权重加载到我们的实现中，该实现可兼容3.45亿、7.62亿及15.42亿参数规模的模型。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37be46de",
   "metadata": {},
   "source": [
    "1.24亿参数的GPT-2模型的配置信息如下:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "0e28e47e",
   "metadata": {},
   "outputs": [],
   "source": [
    "GPT_CONFIG_124M = {\n",
    "    \"vocab_size\": 50257,    # Vocabulary size\n",
    "    \"context_length\": 1024, # Context length\n",
    "    \"emb_dim\": 768,         # Embedding dimension\n",
    "    \"n_heads\": 12,          # Number of attention heads\n",
    "    \"n_layers\": 12,         # Number of layers\n",
    "    \"drop_rate\": 0.1,       # Dropout rate\n",
    "    \"qkv_bias\": False       # Query-Key-Value bias\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1f2d331",
   "metadata": {},
   "source": [
    "- `\"vocab_size\"`表示词表大小为50257个单词，由BPE分词器获得；\n",
    "- `\"context_length\"`代表模型最大输入词元数，由位置编码实现；\n",
    "- `\"emb_dim\"`指词元输入的嵌入维度，将每个输入词元转换为768维向量；\n",
    "- `\"n_heads\"`是多头注意力机制中的注意力头数量；\n",
    "- `\"n_layers\"`是模型内Transformer块的数量；\n",
    "- `\"drop_rate\"`为dropout机制强度，0.1表示训练时随机丢弃10%隐藏单元以防止过拟合；\n",
    "- `\"qkv_bias\"`决定多头注意力机制中的Linear层在计算查询(Q)、键(K)、值(V)张量时是否包含偏置向量。我们将禁用该选项（现代LLM标准做法），但在OpenAI预训练的GPT-2权重时是使用的。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31a62a59",
   "metadata": {},
   "source": [
    "![](./images/scenario.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "cd1a6499",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "\n",
    "class DummyGPTModel(nn.Module):\n",
    "    def __init__(self, cfg):\n",
    "        super().__init__()\n",
    "        self.tok_emb = nn.Embedding(cfg[\"vocab_size\"], cfg[\"emb_dim\"])\n",
    "        self.pos_emb = nn.Embedding(cfg[\"context_length\"], cfg[\"emb_dim\"])\n",
    "        self.drop_emb = nn.Dropout(cfg[\"drop_rate\"])\n",
    "        \n",
    "        # 占位\n",
    "        self.trf_blocks = nn.Sequential(\n",
    "            *[DummyTransformerBlock(cfg) for _ in range(cfg[\"n_layers\"])])\n",
    "        \n",
    "        # 占位\n",
    "        self.final_norm = DummyLayerNorm(cfg[\"emb_dim\"])\n",
    "        self.out_head = nn.Linear(\n",
    "            cfg[\"emb_dim\"], cfg[\"vocab_size\"], bias=False\n",
    "        )\n",
    "\n",
    "    def forward(self, in_idx):\n",
    "        batch_size, seq_len = in_idx.shape\n",
    "        tok_embeds = self.tok_emb(in_idx)\n",
    "        pos_embeds = self.pos_emb(torch.arange(seq_len, device=in_idx.device))\n",
    "        x = tok_embeds + pos_embeds\n",
    "        x = self.drop_emb(x)\n",
    "        x = self.trf_blocks(x)\n",
    "        x = self.final_norm(x)\n",
    "        logits = self.out_head(x)\n",
    "        return logits\n",
    "\n",
    "\n",
    "class DummyTransformerBlock(nn.Module):\n",
    "    def __init__(self, cfg):\n",
    "        super().__init__()\n",
    "        # 占位\n",
    "\n",
    "    def forward(self, x):\n",
    "        # 占位\n",
    "        return x\n",
    "\n",
    "\n",
    "class DummyLayerNorm(nn.Module):\n",
    "    def __init__(self, normalized_shape, eps=1e-5):\n",
    "        super().__init__()\n",
    "        # 占位\n",
    "\n",
    "    def forward(self, x):\n",
    "        # 占位\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f91bb63",
   "metadata": {},
   "source": [
    "![](./images/gpt-2.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "86d5dcd6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[16833,  3626,  6100,   345],\n",
       "        [16833,  1110,   423,  1257]])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import tiktoken\n",
    "\n",
    "tokenizer = tiktoken.get_encoding(\"gpt2\")\n",
    "\n",
    "batch = []\n",
    "txt_1 = \"every effort moves you\"\n",
    "txt_2 = \"every day have fun\"\n",
    "\n",
    "batch.append(torch.tensor(tokenizer.encode(txt_1)))\n",
    "batch.append(torch.tensor(tokenizer.encode(txt_2)))\n",
    "\n",
    "batch = torch.stack(batch, dim=0)\n",
    "\n",
    "batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "43b367cd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "output shape: torch.Size([2, 4, 50257])\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(123)\n",
    "\n",
    "model = DummyGPTModel(GPT_CONFIG_124M)\n",
    "\n",
    "logits = model(batch)\n",
    "print(\"output shape:\", logits.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "f4162e05",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[-1.0327,  0.6600, -0.4785,  ..., -1.5241, -0.3608,  0.7456],\n",
       "         [-0.1192,  0.4539, -0.4432,  ...,  0.2392,  1.3469,  1.2430],\n",
       "         [ 0.5307,  1.6720, -0.4695,  ...,  1.1966,  0.0111,  0.5835],\n",
       "         [ 0.0139,  1.6755, -0.3388,  ...,  1.1586, -0.0435, -1.0400]],\n",
       "\n",
       "        [[-1.0406,  0.1050, -0.6027,  ..., -1.4415,  0.1323,  0.5012],\n",
       "         [-0.7860,  0.5581, -0.0610,  ...,  0.4835, -0.0077,  1.6621],\n",
       "         [ 0.9179,  0.5593, -0.2046,  ...,  0.0988,  0.2858, -0.7168],\n",
       "         [-0.4522,  0.5522, -0.7267,  ...,  1.2022, -0.4960,  0.9700]]],\n",
       "       grad_fn=<UnsafeViewBackward0>)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logits"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11922a07",
   "metadata": {},
   "source": [
    "## 归一化处理"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9b67712",
   "metadata": {},
   "source": [
    "- 归一化层（Layer Normalization，简称LayerNorm，[Ba et al. 2016](https://arxiv.org/abs/1607.06450)）通过将神经网络层的激活值归一化为均值0、方差1的分布，以稳定训练过程并加速模型收敛至有效权重；\n",
    "- 该技术通过减少内部协变量偏移（Internal Covariate Shift）稳定训练动态，显著提升深度网络的收敛速度与泛化能力；\n",
    "- 在Transformer块中，层归一化被应用于多头注意力模块的前后位置（详见后续实现章节），并在最终输出层前再次执行，以保障数据流经各层的分布稳定性。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f17bbce1",
   "metadata": {},
   "source": [
    "![](./images/mlp.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "497c9f57",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.1115,  0.1204, -0.3696, -0.2404, -1.1969],\n",
       "        [ 0.2093, -0.9724, -0.7550,  0.3239, -0.1085]])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.manual_seed(123)\n",
    "\n",
    "batch_example = torch.randn(2, 5)\n",
    "batch_example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "9021bca8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.2260, 0.3470, 0.0000, 0.2216, 0.0000, 0.0000],\n",
       "        [0.2133, 0.2394, 0.0000, 0.5198, 0.3297, 0.0000]],\n",
       "       grad_fn=<ReluBackward0>)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "layer = nn.Sequential(nn.Linear(5, 6), nn.ReLU())\n",
    "output = layer(batch_example)\n",
    "\n",
    "output"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76bf7fb3",
   "metadata": {},
   "source": [
    "该归一化操作独立作用于两个输入行（rows）中的每一个；使用 dim=-1参数意味着计算沿最后一个维度（此处即特征维度）而非行维度进行。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85016d3f",
   "metadata": {},
   "source": [
    "![](./images/mean.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "45bb975d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 6])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "ad184ad9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.1324],\n",
       "        [0.2170]], grad_fn=<MeanBackward1>)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mean = output.mean(dim=1, keepdim=True)\n",
    "mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "8ce54736",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.0231],\n",
       "        [0.0398]], grad_fn=<VarBackward0>)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "var = output.var(dim=-1, keepdim=True)\n",
    "var"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af992ea1",
   "metadata": {},
   "source": [
    "**Z标准化**：减去均值并除以方差平方根（标准差）的操作，能够将输入数据沿列（特征）维度中心化至均值为0、方差为1的分布。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "b098ef02",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.6159,  1.4126, -0.8719,  0.5872, -0.8719, -0.8719],\n",
       "        [-0.0189,  0.1121, -1.0876,  1.5173,  0.5647, -1.0876]],\n",
       "       grad_fn=<DivBackward0>)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output_norm = (output-mean) / torch.sqrt(var)\n",
    "output_norm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "2de9be9a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[9.9341e-09],\n",
       "        [1.9868e-08]], grad_fn=<MeanBackward1>)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "norm_mean = output_norm.mean(dim=-1, keepdim=True)\n",
    "norm_mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "fd6cda42",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[    0.0000],\n",
       "        [    0.0000]], grad_fn=<MeanBackward1>)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.set_printoptions(sci_mode=False)\n",
    "norm_mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "527644b1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1.0000],\n",
       "        [1.0000]], grad_fn=<VarBackward0>)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "norm_var = output_norm.var(dim=-1, keepdim=True)\n",
    "norm_var"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "c93f13da",
   "metadata": {},
   "outputs": [],
   "source": [
    "class LayerNorm(nn.Module):\n",
    "    def __init__(self, emb_dim):\n",
    "        super().__init__()\n",
    "        self.eps = 1e-5\n",
    "        self.scale = nn.Parameter(torch.ones(emb_dim))\n",
    "        self.shift = nn.Parameter(torch.zeros(emb_dim))\n",
    "\n",
    "    def forward(self, x):\n",
    "        mean = x.mean(dim=-1, keepdim=True)\n",
    "        var = x.var(dim=-1, keepdim=True, unbiased=False)\n",
    "        norm_x = (x - mean) / torch.sqrt(var + self.eps)\n",
    "        return self.scale * norm_x + self.shift"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "752074e3",
   "metadata": {},
   "source": [
    "- 缩放与平移\n",
    "    - 需注意，除了通过减去均值并除以方差执行归一化外，我们还添加了两个可学习参数：缩放因子（scale）和平移因子（shift）；\n",
    "    - 初始的缩放因子（乘以1）与平移因子（加上0）并不产生实际影响；然而，这些参数在训练过程中会被大语言模型自动调整（若模型判断此优化能提升其训练任务性能）；\n",
    "    - 这使得模型能够学习最适合其处理数据的缩放与平移方式；\n",
    "    - 另请注意，在计算方差平方根前会添加一个极小值（eps），以避免方差为0时出现除零错误；\n",
    "\n",
    "- ​​有偏方差估计\n",
    "    - 在上述方差计算中，设置unbiased=False意味着使用公式$\\frac{\\sum_i(x_i-\\mu)}{n}$​计算方差（其中n为样本量，此处即特征或列的数量）；该公式未包含贝塞尔校正（Bessel's Correction，即分母使用 n-1），因此提供的是​​有偏方差估计​​；\n",
    "    - 对于嵌入维度n极大的大语言模型而言，使用n与n-1之间的差异可忽略不计；\n",
    "    - 然而，GPT-2在归一化层中训练时采用了有偏方差估计，为兼容后续章节将加载的预训练权重，我们亦沿用此设置；\n",
    "    - 现在让我们实际尝试LayerNorm："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78f635cb",
   "metadata": {},
   "source": [
    "![](./images/framework-1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7da4d21b",
   "metadata": {},
   "source": [
    "## 实现使用GELU激活函数的前馈神经网络"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e8a45cc",
   "metadata": {},
   "source": [
    "- 本节将实现一个用于大语言模型中Transformer块的小型神经网络子模块；\n",
    "- 我们将从激活函数开始介绍；\n",
    "- 深度学习中，ReLU（修正线性单元）激活函数因其简洁性及在各种神经网络架构中的有效性而被广泛使用；\n",
    "- 在大语言模型中，除传统ReLU外还使用多种激活函数，其中值得关注的是GELU（高斯误差线性单元）和SwiGLU（Swish门控线性单元）；\n",
    "- 与简单分段线性的ReLU不同，GELU和SwiGLU是更复杂的平滑激活函数，分别融合高斯函数和Sigmoid门控线性单元，能为深度学习模型提供更优性能GELU([Hendrycks and Gimpel 2016](https://arxiv.org/abs/1606.08415))可通过多种方式实现，其精确版本定义为GELU(x)=x⋅Φ(x)，其中Φ(x)表示标准高斯分布的累积分布函数；\n",
    "- 实践中常采用计算更高效的近似实现：$\\text{GELU}(x) \\approx 0.5 \\cdot x \\cdot \\left(1 + \\tanh\\left[\\sqrt{\\frac{2}{\\pi}} \\cdot \\left(x + 0.044715 \\cdot x^3\\right)\\right]\\right)$ （原始GPT-2模型训练时即采用此近似公式）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "7d4ced38",
   "metadata": {},
   "outputs": [],
   "source": [
    "class GELU(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "\n",
    "    def forward(self, x):\n",
    "        return 0.5 * x * (1 + torch.tanh(\n",
    "            torch.sqrt(torch.tensor(2.0 / torch.pi)) * \n",
    "            (x + 0.044715 * torch.pow(x, 3))\n",
    "        ))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "82a0152a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAxYAAAEiCAYAAABkykQ1AAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvZiW1igAAAAlwSFlzAAAPYQAAD2EBqD+naQAAZ95JREFUeJzt3XlYVGX7B/DvDMuwCYogKCAqKooLIqShuZWKW0Up2aKiZqlh5ZIl/koz36Qyt9ytlCTNfSkzFU1ScwdR0SQXEBc2ZZVlGGbO7w9kEgFl2M6Z4fu5rrned86c5b5nch7uec7zPDJBEAQQERERERFVgVzsAIiIiIiISP+xsCAiIiIioipjYUFERERERFXGwoKIiIiIiKqMhQUREREREVUZCwsiIiIiIqoyFhZERERERFRlLCyIiIiIiKjKWFgQEREREVGVsbAgKsPnn38OmUwmyrVDQ0Mhk8kQHx9f69cuLCzExx9/DBcXF8jlcvj7+9d6DBUh5ntERHXb6NGj0axZM1GuLWbb9ODBA4wbNw6Ojo6QyWSYPHmyKHE8jZjvEbGwqJPi4uIwadIktG7dGhYWFrCwsICHhweCgoJw4cKFEvsW/wMt75GUlAQAiI+Ph0wmw7ffflvudZs1a4YhQ4aU+drZs2chk8kQGhpabXk+TW5uLj7//HNERETU2jUfNW/ePOzatUuUa5dn7dq1mD9/PoYNG4affvoJU6ZMETUeKb5HRIasuGgvfhgbG8PJyQmjR4/GnTt3KnXOiIgIyGQybNu2rdx9ZDIZJk2aVOZr27Ztg0wmq9Xv6rt37+Lzzz9HdHR0rV2zmNhtU3nmzZuH0NBQTJw4EWFhYRg5cqRosUj1PSLAWOwAqHbt2bMHw4cPh7GxMd566y14enpCLpfjypUr2LFjB1auXIm4uDi4urqWOG7lypWwsrIqdb769evXUuTVLzc3F3PmzAEA9O7du8Rrn376KWbMmFGj1583bx6GDRtWqldg5MiReP3116FQKGr0+mX5888/4eTkhEWLFtX6tcsixfeIqC744osv0Lx5c+Tn5+PkyZMIDQ3FsWPHEBMTAzMzM7HDq3F3797FnDlz0KxZM3Tq1KnEa99//z00Gk2NXVvstqk8f/75J5599lnMnj1blOs/SqrvEbGwqFOuX7+O119/Ha6urjh06BAaN25c4vWvv/4aK1asgFxeuiNr2LBhsLOzq61QRWdsbAxjY3H+eRgZGcHIyEiUa6ekpOhFsSjme0RUFwwcOBA+Pj4AgHHjxsHOzg5ff/01fv31V7z22msiRycuExMT0a4tZtuUkpICDw8PUa6tCzHfI+KtUHXKN998g5ycHKxbt65UUQEU/WP84IMP4OLiIkJ0FZOWloaPPvoIHTp0gJWVFaytrTFw4ECcP3++1L75+fn4/PPP0bp1a5iZmaFx48Z49dVXcf36dcTHx8Pe3h4AMGfOHG23/+effw6g9D2a7du3R58+fUpdQ6PRwMnJCcOGDdNu+/bbb9GtWzc0bNgQ5ubm8Pb2LnULgEwmQ05ODn766SfttUePHg2g/PEDK1asQLt27aBQKNCkSRMEBQUhIyOjxD69e/dG+/btcfnyZfTp0wcWFhZwcnLCN99888T3tfhWtsOHD+PSpUvamCIiIrS3MTze5Vx8zKO3r40ePRpWVla4c+cO/P39YWVlBXt7e3z00UdQq9Wl3rslS5agQ4cOMDMzg729PQYMGICzZ89K8j0iqst69OgBoOgHqkdduXIFw4YNg62tLczMzODj44Nff/1VjBBx8+ZNvPfee3B3d4e5uTkaNmyIgICAMsdiZWRkYMqUKWjWrBkUCgWcnZ0xatQo3Lt3DxEREXjmmWcAAGPGjNF+/xR/1z06xkKlUsHW1hZjxowpdY2srCyYmZnho48+AgAUFBRg1qxZ8Pb2ho2NDSwtLdGjRw8cPnxYe4yubRNQNDZu7ty5cHNzg0KhQLNmzTBz5kwolcoS+xXfjnzs2DF06dIFZmZmaNGiBdavX//E97W4DYiLi8Pvv/+ujSk+Pr7c7+Ky2g1dvnurs/2ujfeI/sPCog7Zs2cPWrZsia5du+p8bFpaGu7du1fi8fgfbLXhxo0b2LVrF4YMGYKFCxdi+vTpuHjxInr16oW7d+9q91Or1RgyZAjmzJkDb29vLFiwAB9++CEyMzMRExMDe3t7rFy5EgDwyiuvICwsDGFhYXj11VfLvO7w4cNx5MgR7ZiSYseOHcPdu3fx+uuva7ctWbIEXl5e+OKLLzBv3jwYGxsjICAAv//+u3afsLAwKBQK9OjRQ3vt8ePHl5v3559/jqCgIDRp0gQLFizA0KFDsXr1avTv3x8qlarEvunp6RgwYAA8PT2xYMECtGnTBp988gn++OOPcs9vb2+PsLAwtGnTBs7OztqY2rZtW+4x5VGr1fDz80PDhg3x7bffolevXliwYAHWrFlTYr+3334bkydPhouLC77++mvMmDEDZmZmOHnypCTfI6K6rPgPxwYNGmi3Xbp0Cc8++yz++ecfzJgxAwsWLIClpSX8/f2xc+fOWo/xzJkzOH78OF5//XV89913mDBhAg4dOoTevXsjNzdXu9+DBw/Qo0cPLF26FP3798eSJUswYcIEXLlyBbdv30bbtm3xxRdfAADeffdd7fdPz549S13TxMQEr7zyCnbt2oWCgoISr+3atQtKpVLbPmRlZeGHH35A79698fXXX+Pzzz9Hamoq/Pz8tGM5dG2bgKIepVmzZqFz585YtGgRevXqhZCQkBLtUrFr165h2LBh6NevHxYsWIAGDRpg9OjRuHTpUrnnb9u2LcLCwmBnZ4dOnTppYyr+414XFfnure72uzbeI3qEQHVCZmamAEDw9/cv9Vp6erqQmpqqfeTm5mpfmz17tgCgzIe7u7t2v7i4OAGAMH/+/HJjcHV1FQYPHlzma2fOnBEACOvWrXtiHvn5+YJarS6xLS4uTlAoFMIXX3yh3bZ27VoBgLBw4cJS59BoNIIgCEJqaqoAQJg9e3apfYrzLhYbGysAEJYuXVpiv/fee0+wsrIq8Z49+v8FQRAKCgqE9u3bC88//3yJ7ZaWlkJgYGCpa69bt04AIMTFxQmCIAgpKSmCqamp0L9//xK5L1u2TAAgrF27VrutV69eAgBh/fr12m1KpVJwdHQUhg4dWupaj+vVq5fQrl27EtsOHz4sABAOHz5cYnvxZ/7oZxYYGCgAKPFZCIIgeHl5Cd7e3trnf/75pwBA+OCDD0rFUPz5CII03yMiQ1b8b+vgwYNCamqqcOvWLWHbtm2Cvb29oFAohFu3bmn3feGFF4QOHToI+fn52m0ajUbo1q2b0KpVK+224u+QrVu3lntdAEJQUFCZr23durXM76DHPf7dKwiCcOLEiVL/3mfNmiUAEHbs2FFq/+Lvnye1SYGBgYKrq6v2+f79+wUAwm+//VZiv0GDBgktWrTQPi8sLBSUSmWJfdLT0wUHBwdh7Nix2m26tE3R0dECAGHcuHEl9vvoo48EAMKff/6p3ebq6ioAEI4cOaLdlpKSIigUCmHatGmlrvW4strwx7+Li5XVblT0u7e62+/afI9IENhjUUdkZWUBQJkDsHv37g17e3vtY/ny5aX22b59O8LDw0s81q1bV+NxP06hUGjHgKjVaty/fx9WVlZwd3dHVFRUiXjt7Ozw/vvvlzpHZaaha926NTp16oTNmzdrt6nVamzbtg0vvvgizM3Ntdsf/f/p6enIzMxEjx49SsSni4MHD6KgoACTJ08uMf7lnXfegbW1dYmeEKDoMx4xYoT2uampKbp06YIbN25U6vqVMWHChBLPe/ToUeL627dvh0wmK3MQYGU+H318j4ikrG/fvrC3t4eLiwuGDRsGS0tL/Prrr3B2dgZQ1Iv9559/4rXXXkN2dra2J/v+/fvw8/PD1atXKz2LVGU9+t2rUqlw//59tGzZEvXr1y/VPnh6euKVV14pdY7KfP88//zzsLOzK9E+pKenIzw8HMOHD9duMzIygqmpKYCiW0HT0tJQWFgIHx+fSrcPe/fuBQBMnTq1xPZp06YBQKnvPg8PD+1tbUBRD4m7u3utffdV5Lu3uttvfXuP9B1Ht9QR9erVA1DUBfy41atXIzs7G8nJySX+wT+qZ8+etTJ4+2lfGsX35a9YsQJxcXEl7ttv2LCh9v9fv34d7u7u1TqAa/jw4Zg5cybu3LkDJycnREREICUlpUTDARTdcva///0P0dHRJe7frOy82jdv3gQAuLu7l9huamqKFi1aaF8v5uzsXOpaDRo0KDWVcE0pHi/x+PXT09O1z69fv44mTZrA1ta2Wq6pb+8RkdQtX74crVu3RmZmJtauXYsjR46UmIXt2rVrEAQBn332GT777LMyz5GSkgInJ6dqi+lp36F5eXkICQnBunXrcOfOHQiCoH0tMzNT+/+vX7+OoUOHVltcxsbGGDp0KDZu3AilUgmFQoEdO3ZApVKVah9++uknLFiwAFeuXClxi2bz5s0rde2bN29CLpejZcuWJbY7Ojqifv36pb77mjZtWuocj38/16SKfPdWd/utb++RvmNhUUfY2NigcePGiImJKfVa8ZiLml5szMzMDHl5eWW+Vnz/69OmMZw3bx4+++wzjB07FnPnzoWtrS3kcjkmT55co9P/AUWFRXBwMLZu3YrJkydjy5YtsLGxwYABA7T7HD16FC+99BJ69uyJFStWoHHjxjAxMcG6deuwcePGGo2vWHmzJT3ayOqivMb88cHYT7u+lFT3e0RkaLp06aKdFcrf3x/PPfcc3nzzTcTGxsLKykr7ffvRRx/Bz8+vzHM8/ofckygUiiq3D++//z7WrVuHyZMnw9fXFzY2NpDJZHj99ddrvH14/fXXsXr1avzxxx/w9/fHli1b0KZNG3h6emr3+fnnnzF69Gj4+/tj+vTpaNSoEYyMjBASElJqULyuKvrDlVTbh9r47hXrPaprWFjUIYMHD8YPP/yA06dPo0uXLrV+fVdXV1y+fLnM12JjY7X7PMm2bdvQp08f/PjjjyW2Z2RklOhRcXNzw6lTp6BSqcqdGlDXHoTmzZujS5cu2Lx5MyZNmoQdO3bA39+/xK9427dvh5mZGfbv319ie1m3jVX0+sXvSWxsLFq0aKHdXlBQgLi4OPTt21enPHRVPFjz8cH6j//Kows3Nzfs378faWlpT+y10Jf3iMiQFf/x26dPHyxbtgwzZszQ/jszMTGpln9frq6u2nbgcbq0D4GBgViwYIF2W35+fqnvLjc3tzJ/ZHuUru1Dz5490bhxY2zevBnPPfcc/vzzT/zf//1fqfhatGiBHTt2lDj/47eE6nJtV1dXaDQaXL16tcRkG8nJycjIyHjqe1ZVNdU+VGf7LfZ7VNdwjEUd8vHHH8PCwgJjx45FcnJyqddruhofNGgQbt++XWolZaVSiR9++AGNGjVC586dn3gOIyOjUnFu3bq11L28Q4cOxb1797Bs2bJS5yg+3sLCAkDpL8QnGT58OE6ePIm1a9fi3r17pbq5jYyMIJPJSvxaEx8fX+bq0ZaWlhW6dt++fWFqaorvvvuuRO4//vgjMjMzMXjw4ArHXxmurq4wMjLCkSNHSmxfsWJFpc85dOhQCIKgXeDoUY/mqC/vEZGh6927N7p06YLFixcjPz8fjRo1Qu/evbF69WokJiaW2j81NVWn8w8aNAgnT55EZGRkie0ZGRnYsGEDOnXqBEdHxyeeo6z2YenSpaV+PR86dCjOnz9f5sxVxcdbWlpqr18Rcrkcw4YNw2+//YawsDAUFhaW2T48eg0AOHXqFE6cOFFiP13apkGDBgEAFi9eXGL7woULAaDGv/vc3NwAoET7oFarS80CqIvqbr/Ffo/qGvZY1CGtWrXCxo0b8cYbb8Dd3V278rYgCIiLi8PGjRshl8u1g/MetW3btjIHfvfr1w8ODg7a54cOHUJ+fn6p/fz9/fHuu+9i7dq1CAgIwNixY+Hl5YX79+9j8+bNiImJwfr167UD28ozZMgQfPHFFxgzZgy6deuGixcvYsOGDSV+pQaAUaNGYf369Zg6dSpOnz6NHj16ICcnBwcPHsR7772Hl19+Gebm5vDw8MDmzZvRunVr2Nraon379mjfvn2513/ttdfw0Ucf4aOPPoKtrW2pX+oGDx6MhQsXYsCAAXjzzTeRkpKC5cuXo2XLlqXu3/f29sbBgwexcOFCNGnSBM2bNy9zKmB7e3sEBwdjzpw5GDBgAF566SXExsZixYoVeOaZZ8odF1NdbGxsEBAQgKVLl0Imk8HNzQ179uxBSkpKpc/Zp08fjBw5Et999x2uXr2KAQMGQKPR4OjRo+jTpw8mTZoEQH/eI6K6YPr06QgICEBoaCgmTJiA5cuX47nnnkOHDh3wzjvvoEWLFkhOTsaJEydw+/btUusLbd++HVeuXCl13sDAQMyYMQNbt25Fz549MX78eLRp0wZ3795FaGgoEhMTKzRZyJAhQxAWFgYbGxt4eHjgxIkTOHjwYInxd8V5bNu2TdsWeXt7Iy0tDb/++itWrVoFT09PuLm5oX79+li1ahXq1asHS0tLdO3a9YljIYYPH46lS5di9uzZ6NChQ6npuocMGYIdO3bglVdeweDBgxEXF4dVq1bBw8OjxPhHXdomT09PBAYGYs2aNcjIyECvXr1w+vRp/PTTT/D39y9z/aXq1K5dOzz77LMIDg7W9kBv2rQJhYWFlT5ndbffYr9HdU4tz0JFEnDt2jVh4sSJQsuWLQUzMzPB3NxcaNOmjTBhwgQhOjq6xL5Pmm4Wj0wlVzz1aHmPsLAwQRCKptabMmWK0Lx5c8HExESwtrYW+vTpI/zxxx8Vij0/P1+YNm2a0LhxY8Hc3Fzo3r27cOLECaFXr15Cr169Suybm5sr/N///Z/2Wo6OjsKwYcOE69eva/c5fvy44O3tLZiampaYuu7x6eoe1b179zKnriv2448/Cq1atRIUCoXQpk0bYd26dWWe78qVK0LPnj0Fc3NzAYB2WtXypu9btmyZ0KZNG8HExERwcHAQJk6cKKSnp5fYp6zpYgWh9PSI5Snv+NTUVGHo0KGChYWF0KBBA2H8+PFCTExMmdPNWlpaljq+rPwLCwuF+fPnC23atBFMTU0Fe3t7YeDAgUJkZKR2Hym+R0SGrPjf1pkzZ0q9plarBTc3N8HNzU0oLCwUBEEQrl+/LowaNUpwdHQUTExMBCcnJ2HIkCHCtm3btMcVTz1a3uPo0aOCIAjC7du3hXHjxglOTk6CsbGxYGtrKwwZMkQ4efJkhWJPT08XxowZI9jZ2QlWVlaCn5+fcOXKFcHV1bXUtNX3798XJk2aJDg5OQmmpqaCs7OzEBgYKNy7d0+7z+7duwUPDw/B2Ni4xHdded8VGo1GcHFxEQAI//vf/8p8fd68eYKrq6ugUCgELy8vYc+ePWWeT5e2SaVSCXPmzNG2dS4uLkJwcHCJaYAFofwp38tqP8tS3vHXr18X+vbtKygUCsHBwUGYOXOmEB4eXuZ0sxX97q3u9ru23iMSBJkgcDQKERERERFVDcdYEBERERFRlbGwICIiIiKiKmNhQUREREREVcbCgoiIiIiIqoyFBRERERERVRkLCyIiIiIiqrI6t0CeRqPB3bt3Ua9ePZ2WhCciMmSCICA7OxtNmjSBXF53f3NiG0FEVJIu7UOdKyzu3r0LFxcXscMgIpKkW7duwdnZWewwRMM2goiobBVpH+pcYVGvXj0ARW+OtbW1TseqVCocOHAA/fv3h4mJSU2EVysMIQ/mIB2GkIch5ABULY+srCy4uLhovyPrqrreRjAH6TCEPAwhB8Aw8qit9qHOFRbFXdvW1taVajQsLCxgbW2tt/9hAYaRB3OQDkPIwxByAKonj7p++09dbyOYg3QYQh6GkANgGHnUVvtQd2+kJSIiIiKiasPCgoiIiIiIqkzUwmLlypXo2LGjtsvZ19cXf/zxxxOP2bp1K9q0aQMzMzN06NABe/furaVoiYiotrB9ICLSP6IWFs7Ozvjqq68QGRmJs2fP4vnnn8fLL7+MS5culbn/8ePH8cYbb+Dtt9/GuXPn4O/vD39/f8TExNRy5EREVJPYPhAR6R9RC4sXX3wRgwYNQqtWrdC6dWt8+eWXsLKywsmTJ8vcf8mSJRgwYACmT5+Otm3bYu7cuejcuTOWLVtWy5ETEVFNYvtARKR/JDMrlFqtxtatW5GTkwNfX98y9zlx4gSmTp1aYpufnx927dpV7nmVSiWUSqX2eVZWFoCi0fEqlUqnGIv31/U4qTGEPJiDdBhCHgaRg1qDL/ZcRmt15fKQcu411T4QEdUVR6/ew593ZRgoCDV6HdELi4sXL8LX1xf5+fmwsrLCzp074eHhUea+SUlJcHBwKLHNwcEBSUlJ5Z4/JCQEc+bMKbX9wIEDsLCwqFTM4eHhlTpOagwhD+YgHYaQhz7nsOWGHH8ny9FQYQQb03AY69gfnZubWzOBVUFNtw8Af3x6HHOQDkPIwxByAPQ/j5tpuZi85QKy8o3gcyYBr3dx1el4XfIWvbBwd3dHdHQ0MjMzsW3bNgQGBuKvv/4qt/HQVXBwcIlfsYoX+ejfv3+l5igPDw9Hv3799HYeY8Aw8mAO0mEIeeh7Dj+fSsDfJ65ABuCVZhoM9NM9j+I/qKWkptsHgD8+lYc5SIch5GEIOQD6mYdSDSyKMUJWvgyuVgIsUi5h796yx6qVR5cfnkQvLExNTdGyZUsAgLe3N86cOYMlS5Zg9erVpfZ1dHREcnJyiW3JyclwdHQs9/wKhQIKhaLUdhMTk0r/AVGVY6XEEPJgDtJhCHnoYw5Hr6bif3tjAQDT+rWCy4N/KpWHFPOu6fYB4I9Pj2MO0mEIeRhCDoD+5iEIAiZvuYDE3GQ0tDTF2Na5Nf7Dk+iFxeM0Gk2JbulH+fr64tChQ5g8ebJ2W3h4eLn33BIRGbIbqQ8QtCEKao2AVzs74d0ezfDHH/+IHVaNqYn2gT8+lY05SIch5GEIOQD6l8eqv65jb0wyjOUyLHvDEymXTtT4D0+iFhbBwcEYOHAgmjZtiuzsbGzcuBERERHYv38/AGDUqFFwcnJCSEgIAODDDz9Er169sGDBAgwePBibNm3C2bNnsWbNGjHTICKqdZm5Koz76Syy8gvRuWl9zHulA2TQiB1WtWH7QERUeUf+TcU3+64AAGa/1A4+rg2g4x1QlSJqYZGSkoJRo0YhMTERNjY26NixI/bv349+/foBABISEiCX/zcCsVu3bti4cSM+/fRTzJw5E61atcKuXbvQvn17sVIgIqp1hWoNJv0ShRv3ctDExgyrR/rAzMQIKpXhFBZsH4iIKifhfi7e/+UcNAIQ4O2MEV2borCwsFauLWph8eOPPz7x9YiIiFLbAgICEBAQUEMRERFJ3/9+/wdHr96DuYkRvg/0gX290rfy6Du2D0REusstKMS7YWeRmaeCp0t9zPVvD5lMVmvXF3WBPCIi0s3GUwkIPR4PAFg03BPtmtiIGxAREUmCIAj4ZPtFXEnKhp2VKVaN6AwzE6NajYGFBRGRnjhx/T5m7Y4BAEzr1xoD2jcWOSIiIpKKH47G4bfzd2Esl2HFW95obGNe6zGwsCAi0gMJ93MxcUMkCjUCXvRsgknPtxQ7JCIikohjV+8h5OGsgJ8N8UCX5raixMHCgohI4rLzVRi3/gwyclXo6GyD+cM61uo9s0REJF230nIx6ZcoaARgmLczRvnqtrJ2dWJhQUQkYWqNgMmbovFv8gM4WCvw/SifWr9nloiIpCmvQI3xYZHaH57+V8uDtR/HwoKISMLm74/FoSspUBjLsWakDxyszcQOiYiIJEAQBMzYcQGXE7PQ0NIUq0Z4i/7DEwsLIiKJ2hF1G6v+ug4A+GZYR3i61Bc3ICIikowfj8Vhd/RdGMllWP5WZzSpX/uDtR/HwoKISILOJaRjxo6LAICgPm54uZOTyBEREZFUHL92DyF/FK2s/engtni2RUORIyrCwoKISGISM/PwblgkCgo16OfhgGn93MUOiYiIJOJ2ei4m/XIOao2AVzs7YXS3ZmKHpMXCgohIQvJVary7PhKp2Uq0cayHxcM7QS7nDFBERFTURowPi0RaTgHaO1lj3isdJDVLIAsLIiKJEAQB07ddwMU7mbC1NMX3o3xgqTAWOywiIpIAQRAwc8dFXLqbBVuJDNZ+HAsLIiKJWBFx/ZFVUzvDxdZC7JCIiEgiQo/HY8e5OzCSy7DsTS84N5BeG8HCgohIAsIvJ+PbA7EAgDkvt5PMQDwiIhLfyRv38b/fi1bWnjmoLbq52YkcUdlYWBARiSw2KRuTN52DIACjfF3xVlfxVk0lIiJpuZORh6ANUVBrBPh3aoKx3ZuJHVK5WFgQEYkoPacA49afQU6BGr4tGuKzIR5ih0RERBKRr1Jj4s+RuJ9TAI/G1gh5taOkBms/joUFEZFIVGoN3tsQhVtpeXCxNceKtzrDxIhfy0REVDRY+/92xuDC7Uw0sDDB6pHeMDeV1mDtx7EFIyISyf/2XMaJG/dhaWqEH0Y9gwaWpmKHREREErH+xE1sj7oNuQxY9qZ+TOjBwoKISAS/nE7ATyduAgAWDe8Ed8d6IkdERERScerGfczdcxkAEDywLbq3lOZg7ceJWliEhITgmWeeQb169dCoUSP4+/sjNjb2iceEhoZCJpOVeJiZmdVSxEREVXcmPg2zdscAAD7q3xr92zmKHBEREUlFYmYegjZGoVAj4CXPJhjXo7nYIVWYqIXFX3/9haCgIJw8eRLh4eFQqVTo378/cnJynnictbU1EhMTtY+bN2/WUsRERFVzJyMPE8IioVILGNyxMYL6tBQ7JCIikoh8lRoTwiJx70EB2ja2xtdDpT1Y+3GiFhb79u3D6NGj0a5dO3h6eiI0NBQJCQmIjIx84nEymQyOjo7ah4ODQy1FTERUeXkFaowPO6ud3WP+MP1qMGoTe7SJqK4RBAGf7YrB+duZsDE3weoR0h+s/ThJjbHIzMwEANja2j5xvwcPHsDV1RUuLi54+eWXcenSpdoIj4io0gRBwCfbLyDmThZsLU2xZpQ3LEyNxQ5LstijTUR1zc+nErA1sniwtheaNpT+YO3HSaZV02g0mDx5Mrp374727duXu5+7uzvWrl2Ljh07IjMzE99++y26deuGS5cuwdnZudT+SqUSSqVS+zwrKwsAoFKpoFKpdIqxeH9dj5MaQ8iDOUiHIeRRGzmsORqHX8/fhbFchu+Gd4SDlUm1X68qeUjt89u3b1+J56GhoWjUqBEiIyPRs2fPco8r7tEmItInZ+LTMOfXoh/KPxnQBj1a2YscUeVIprAICgpCTEwMjh079sT9fH194evrq33erVs3tG3bFqtXr8bcuXNL7R8SEoI5c+aU2n7gwAFYWFSuEgwPD6/UcVJjCHkwB+kwhDxqKofL6TKsuSIHIIO/ayHu/3MSe/+pkUsBqFweubm5NRBJ9dG1R1uj0aBz586YN28e2rVrVxshEhFVSnJWPt7bUDRYe3DHxni3ZwuxQ6o0SRQWkyZNwp49e3DkyJEyex2exMTEBF5eXrh27VqZrwcHB2Pq1Kna51lZWXBxcUH//v1hbW2t07VUKhXCw8PRr18/mJiY6HSslBhCHsxBOgwhj5rMIe5eDj5dfQoCCjHcxxlzX2pbY+MqqpJHcW+uFNVUjzbAXu3HMQfpMIQ8DCEHoGbzUBZqMD7sLFKzlXB3sMKXL7VFYWFhtV+ntnq0RS0sBEHA+++/j507dyIiIgLNm+s+nZZarcbFixcxaNCgMl9XKBRQKBSltpuYmFT6D4iqHCslhpAHc5AOQ8ijunPIzldh4sZoZOcXwse1Aeb6d4Cpcc0PbatMHlL+7GqqRxtgr3Z5mIN0GEIehpADUDN5bLouR3SKHBZGAl5rkoG/Dh2o9ms8qqZ7tEUtLIKCgrBx40bs3r0b9erVQ1JSEgDAxsYG5ubmAIBRo0bByckJISEhAIAvvvgCzz77LFq2bImMjAzMnz8fN2/exLhx40TLg4jocRqNgCmbo3E9NQeNbcywcoR3rRQVhqYme7QB9mo/jjlIhyHkYQg5ADWXx6Yzt3HixGXIZMCyt7zRo1XNLYJXWz3aohYWK1euBAD07t27xPZ169Zh9OjRAICEhATI5f81xunp6XjnnXeQlJSEBg0awNvbG8ePH4eHh0dthU1E9FSLDv6Lg/+kQGEsx+qR3rCvV7rnlMpXGz3aAHu1y8McpMMQ8jCEHIDqzSPyZjq++L1osN10P3c879G4Ws77NDXdoy36rVBPExERUeL5okWLsGjRohqKiIio6v64mIilfxb9Sh7yagd0dK4vbkB6iD3aRGSokrPyMfHnooVSB3VwxMRebmKHVG0kMXibiMhQXEnKwrSt5wEAbz/XHK921u32HSrCHm0iMkQFhRpM/DkSKdlKtHawwvxhnga1UCoLCyKiapKRW4B310cit0CNbm4NETywjdgh6S32aBORIZrz2yVEJWTA2swYa0b6wFJhWH+KcyQhEVE1UGsEvP/LOSSk5cK5gTmWvdkZxkb8iiUioiKbTidgw6kEyGTAkte90MzOUuyQqh1bPSKiajB/fyyOXr0HMxM51oz0ga2lqdghERGRREQlpGPW7qKVtT/q744+bRqJHFHNYGFBRFRFey7cxaq/rgMA5g/zhEcT3aYpJSIiw5WSXTRYu0CtwYB2jnivt+EM1n4cCwsioir4JzEL07deAACM79UCL3o2ETkiIiKSioJCDYI2RCE5S4lWjazw7WuGNVj7cSwsiIgqKSO3AOPDIpGnUqNHKzt87MfB2kRE9J+5ey7jTHw66imMsXqkN6wMbLD241hYEBFVgloj4INN0UhIy4WLrTmWvuEFI7nh/gpFRES62XLmFsJO3iwarP1GJ7SwtxI7pBrHwoKIqBIWHIjFkX9TYWYix+oRPqhvwcHaRERUJPpWBj7dFQMAmNK3NZ5v4yByRLWDhQURkY7+uJiIFRFFg7W/HtqRg7WJiEgrNVuJCWFFg7X7ezhgUp+WYodUa1hYEBHp4GpyNj56uLL2uOea4+VOTiJHREREUqFSFw3WTsrKh5u9JRa85gl5HbpNloUFEVEFZeWrMD4sEjkPV9aewZW1iYjoEV/+/g9Ox6fBSmGMNaN8UM/MROyQahULCyKiCtBoBEzdfB437uXAqX7RYG2urE1ERMW2Rd5G6PF4AMCi4Z3gVgcGaz+OrSIRUQUsO3wNB/9JhqmxHCtHdEZDK4XYIRERkURcuJ2BmTsvAgAm922Ffh51Y7D241hYEBE9xeErKVh08F8AwP/826Ojc31xAyIiIsm49+DhYO1CDfq2bYQPnm8ldkiiYWFBRPQEN+/n4MNN5yAIwFtdm+I1HxexQyIiIokoHqx9NzMfLewtsXB4pzo1WPtxLCyIiMqRV6DGhJ+jkJVfCK+m9THrRQ+xQyIiIgmZt/cfnIp7OFh7pA+s69hg7cexsCAiKoMgCJi58yL+ScyCnZUpVr7lDYWxkdhhERGRROyIuo11f8cDABa85omWjereYO3HsbAgIirD+hM3sfPcHRjJZVj2Zmc42piJHRIREUlEzJ1MBO8oGqz9wfMt4dfOUeSIpEHUwiIkJATPPPMM6tWrh0aNGsHf3x+xsbFPPW7r1q1o06YNzMzM0KFDB+zdu7cWoiWiuiLyZhrm7rkMAAge2AbPtmgockRERCQV9x8oMT4sEspCDV5o0wiT+7YWOyTJELWw+OuvvxAUFISTJ08iPDwcKpUK/fv3R05OTrnHHD9+HG+88QbefvttnDt3Dv7+/vD390dMTEwtRk5EhiolOx/vbYhCoUbA4I6N8fZzzcUOiYiIJKJQrcGkjedwJyMPze04WPtxxmJefN++fSWeh4aGolGjRoiMjETPnj3LPGbJkiUYMGAApk+fDgCYO3cuwsPDsWzZMqxatarGYyYiw6V62GAkZynRqpEVvhnaETIZGwwiIioS8scVnLhxH5amRlg90hs25nV7sPbjRC0sHpeZmQkAsLW1LXefEydOYOrUqSW2+fn5YdeuXWXur1QqoVQqtc+zsrIAACqVCiqVSqf4ivfX9TipMYQ8mIN0GEIexbF/sy8Wp+PSYKkwwtLXPWEqF/Qqr6p8FlLLMyQkBDt27MCVK1dgbm6Obt264euvv4a7u/sTj9u6dSs+++wzxMfHo1WrVvj6668xaNCgWoqaiAzZ7ui7+PFYHICiwdqtHeqJHJH0SKaw0Gg0mDx5Mrp374727duXu19SUhIcHEquZujg4ICkpKQy9w8JCcGcOXNKbT9w4AAsLCwqFWt4eHiljpMaQ8iDOUiHvudx7r4Mof/eAgAMdy1A7Jm/8PQRX9JUmc8iNze3BiKpvOJbZZ955hkUFhZi5syZ6N+/Py5fvgxLS8syjym+VTYkJARDhgzBxo0b4e/vj6ioqCe2K0RET3M7B/hud9HYu0l9WmJA+8YiRyRNkiksgoKCEBMTg2PHjlXreYODg0v0cGRlZcHFxQX9+/eHtbW1TudSqVQIDw9Hv379YGKiv11fhpAHc5AOQ8gjNjEDH686BQAY91wzfOKnnwPxqvJZFPfmSgVvlSUiqUjLKcCPsUZQFmrQ290eU/rpZxtRGyRRWEyaNAl79uzBkSNH4Ozs/MR9HR0dkZycXGJbcnIyHB3LnuZLoVBAoVCU2m5iYlLpP4KqcqyUGEIezEE69DWPHGUhJm+9BKVGhi7NGmDGwLYwNtLvmbgr81lI/bOriVtliYieplCtwZQtF5CmlKGprTmWDPeCEQdrl0vUwkIQBLz//vvYuXMnIiIi0Lz502df8fX1xaFDhzB58mTttvDwcPj6+tZgpERkiARBwIwdF3EtNQfWJgIWv9ZR74sKQ1RTt8oCHIf3OOYgHYaQhyHk8NW+WBy/kQZTuYClr7WHhYl+5lNbY/BELSyCgoKwceNG7N69G/Xq1dN++dvY2MDc3BwAMGrUKDg5OSEkJAQA8OGHH6JXr15YsGABBg8ejE2bNuHs2bNYs2aNaHkQkX766Xg8fjt/F8ZyGca0LoR9vdK9myS+mrpVFuA4vPIwB+kwhDz0NYeoezL8dNUIAPBWSw3iz59A/HmRg6qimh6DJ2phsXLlSgBA7969S2xft24dRo8eDQBISEiAXP7fL4jdunXDxo0b8emnn2LmzJlo1aoVdu3axYF5RKSTqIR0fLn3HwDAx36t4ZBxSeSIqCw1easswHF4j2MO0mEIeehzDv8kZuOT708B0GBc96booLmhl3kUq60xeKLfCvU0ERERpbYFBAQgICCgBiIiorrg/gMlgjZEQaUWMLhDY4z2bYo//mBhISW1dassx+GVjTlIhyHkoW85pOcUIGhTNPJVGvRoZYeP+rtj/74bepdHWWp6DJ4kBm8TEdUWtUbA5M3RSMzMRwt7S3w1tAO4Bp708FZZIhJDoVqDDzadw620PDS1tcDSNzhYWxccpUhEdcqSQ1dx9Oo9mJsYYdUIb9Qz0+9fnwzVypUrkZmZid69e6Nx48bax+bNm7X7JCQkIDExUfu8+FbZNWvWwNPTE9u2beOtskSkk/kHYrVtxOqR3qhvYSp2SHqlUj0WcXFxOHr0KG7evInc3FzY29vDy8sLvr6+MDMzq+4YiYiqRURsCpb+eRUAMO/V9lw1VcJ4qywR1bY9F+5i9V83AADzAzqibWPdxlmRjoXFhg0bsGTJEpw9exYODg5o0qQJzM3NkZaWhuvXr8PMzAxvvfUWPvnkE7i6utZUzEREOruTkYfJm6MhCMBbXZviFa8nDwQmIqK645/ELEzfegEAML5nCwzp2ETkiPRThQsLLy8vmJqaYvTo0di+fTtcXFxKvK5UKnHixAls2rQJPj4+WLFiBX81IiJJKCjU4L0NUcjIVaGjsw1mveghdkgGjb3aRKRPMnILMD4sEnkqNXq0ssPHA9qIHZLeqnBh8dVXX8HPz6/c1xUKBXr37o3evXvjyy+/RHx8fHXER0RUZfP2/oPztzJgY26C5W92hsLYSOyQDBJ7tYlI36g1Aj7YFI2EtFw4NzDHd69zsHZVVLiweFJR8biGDRuiYcOGlQqIiKg6/X4hEaHH4wEAC1/zhItt5RY9oydjrzYR6aMFB2Jx5N9UmJnIsXqkNxpYcrB2VVRqVqjQ0NAytxcWFiI4OLgq8RARVZsbqQ/wyfaie2Yn9nbDC20dRI7IcH311Vc4deoU3nvvvVJFBfBfr/aqVatw5coVtGjRQoQoiYj+s/diIlZEXAcAfD20I9o1sRE5Iv1XqcLigw8+QEBAANLT07XbYmNj0bVrV/zyyy/VFhwRUWXlFajx3oYoPFAWoktzW0zr11rskAyarr3a3t7eNRgNEdGTxSZl46Ot5wEA7/Rojpc7OYkckWGoVGFx7tw53L59Gx06dEB4eDiWL1+Ozp07o02bNjh//nx1x0hEpLPZv8bgSlI27KxMsewNLxgbcdme2sJebSKSssxcFcaHnUVugRrd3BriEw7WrjaVamnd3Nzw999/49VXX8WAAQMwZcoU/PDDD9iwYQNsbNiNRETi2nr2FracvQ25DPjudS80suZMRLWJvdpEJFVqjYAPN59D/P1cONU3x7I3O/OHp2pU6Xfy999/x6ZNm+Dr64v69evjxx9/xN27d6szNiIincUmZeOz3TEAgCl9W6NbSzuRI6p72KtNRFK1KPxfRMSmQmFcNFjbloO1q1WlCovx48cjICAAn3zyCY4ePYoLFy7A1NQUHTp0wJYtW6o7RiKiCslRFmLihkjkqzTo2doeQX1aih1SncRebSKSon0xiVh2+BoA4KuhHdDeid9H1a1ShcXff/+NU6dOYdq0aZDJZHB0dMTevXvxxRdfYOzYsdUdIxHRUwmCgJk7L+JGag4crc2weHgnyDkXuWjYq01EUnI1ORvTthT1mI7t3hyveDmLHJFhqlRhERkZCU9Pz1Lbg4KCEBkZWeWgiIh09cvpW9gdfRdGchmWvenF7m0RsVebiKQkM0+Fd8MikVOgxrMtbBE8iIO1a0qFF8h7lEKhKPc1d3f3SgdDRFQZMXcy8flvlwAAH/u5w6eZrcgR1W3FvdrFP0AV92ovX74cY8eOxWuvvSZyhERUV2g0AqZsjkbcvRw0sTHD8jc7w4SDtWtMhd/ZAQMG4OTJk0/dLzs7G19//TWWL19epcCIiCoiO1+FSRujUFCowQttGuGdHlx4TWzs1SYiqVh86Cr+vJLycLC2Dxpalf/jOFVdhXssAgICMHToUNjY2ODFF1+Ej48PmjRpAjMzM6Snp+Py5cs4duwY9u7di8GDB2P+/Pk1GTcREQRBwIwdF7XTBi54zZPjKiSAvdpEJAX7LyXhu0NXAQDzXumADs4crF3TKtxj8fbbb+PGjRuYOXMmLl++jHfffRc9evTAM888Az8/P3z//fdo2rQpzpw5g82bN6Np06ZPPeeRI0fw4osvokmTJpDJZNi1a9cT94+IiIBMJiv1SEpKqmgaRGRAfj55E79fSISxXIalb3qhvgXHVYiFvdpEJCXXUv4brD26WzMM9eZg7dqg0xgLhUKBESNGYMSIEQCAzMxM5OXloWHDhjAxMdH54jk5OfD09MTYsWPx6quvVvi42NhYWFtba583atRI52sTkX67eDsTc/f8AwCYMbANOjdtIHJEdRt7tYlIKrLyiwZrP1AWomtzW/zf4LZih1RnVGrwdjEbG5sqzUk+cOBADBw4UOfjGjVqhPr161f6ukSk37LyVQjaGIUCtQb9PBzw9nPNxQ6pznv77bcxYsQIbN26FZs3b8aaNWuQmZkJAJDJZPDw8ICfnx/OnDmDtm3ZyBNRzdBoBEzdHI0bqTlobGOG5W9xsHZt0qmw+O6778rcbmNjg9atW8PX17dagnqaTp06QalUon379vj888/RvXv3cvdVKpVQKpXa51lZWQAAlUoFlUql03WL99f1OKkxhDyYg3TUdh6CIODjrReQkJYLp/pmCPH3QGFhYZXOyc+ienKv7l5tIiJdfffnVRz8JwWmxnKsGuENOw7WrlU6FRaLFi0qc3tGRgYyMzPRrVs3/Prrr7C1rZmpHhs3boxVq1bBx8cHSqUSP/zwA3r37o1Tp06hc+fOZR4TEhKCOXPmlNp+4MABWFhYVCqO8PDwSh0nNYaQB3OQjtrK42iSDPvijGAkEzDc+QH+Plx9163Ln0Vubm61x1HVXm0iIl2EX07G4oNFg7W/9G8PT5f64gZUB+lUWMTFxZX72o0bNzBixAh8+umnWLFiRZUDK4u7u3uJGUW6deuG69evY9GiRQgLCyvzmODgYEydOlX7PCsrCy4uLujfv3+JcRoVoVKpEB4ejn79+un1r2+GkAdzkI7azOPS3Sx8tOYUAAGfDGiDMd1cq+W8/Cz+682tiuru1T5y5Ajmz5+PyMhIJCYmYufOnfD39y93/4iICPTp06fU9sTERDg6Oup0bSLSL9dTH2Dq5mgAQKCvKwJ8XMQNqI6q0hiLR7Vo0QJfffUVxo4dW12nrJAuXbrg2LFj5b6uUCjKnPrQxMSk0n9AVOVYKTGEPJiDdNR0Hln5Kny45QJUagF92zrgnZ5ukMmqd2rZuvxZVEfe1d2rzQk+iKgisvNVeHf9WWQrC9GlmS0+HeIhdkh1VrUVFgDQtGnTWp/6NTo6Go0bN67VaxJR7RIEAcHbL+Lmw/Uqvg3oWO1FBVVddfdqc4IPInoajUbAtC3ncT01B47WZlj2lhcHa4uoWguLixcvwtW14rcmPHjwANeuXdM+j4uLQ3R0NGxtbdG0aVMEBwfjzp07WL9+PQBg8eLFaN68Odq1a4f8/Hz88MMP+PPPP3HgwIHqTIOIJObnUwn4/WLRehXLuF6FXqrNXm1dJvggIv22/PA1HLicDFMjOVaN9EajemZih1Sn6VRYlHcPbmZmJiIjIzFt2jQEBgZW+Hxnz54tcT9s8ViIwMBAhIaGIjExEQkJCdrXCwoKMG3aNNy5cwcWFhbo2LEjDh48WOY9tURkGGLuZGLub5cBAJ8MaAMvrleht2q6V7syE3xw5sCSmIN0GEIeNZ3D4dhULDz4LwDg8xfbop2jZY1cq65/Froco1NhUb9+/XJvP5DJZBg3bhxmzJhR4fP17t0bgiCU+3poaGiJ5x9//DE+/vjjCp+fiPRbdr4Kkx6uV/FCm0YY14PrVegzXXu1dVWZCT44c2DZmIN0GEIeNZFDSh6w8KIRBEGG7g4aWCafx96956v9Oo+qq5+FLrMG6lRYHD58uMzt1tbWaNWqFczMzJCSkoImTZrocloiolIEQcDMnTGIv5+LJjZm+DbAk+MqJK66e7Wrw9Mm+ODMgSUxB+kwhDxqKocHykIErD6FPHUOvJvWx5oxPjA1rrlxFXX9s9Bl1kCdCotevXo98fXz58+jc+fOUKvVupyWiKiUX07fwm/n78JILsPSN73QwJLjKqSuunu1q8PTJvjgzIFlYw7SYQh5VGcOgiAgeNMFXEvNgYO1AitHesPSvHYWwaurn4Uu+1fr4G0iourwT2IW5vx2CQAw3c8d3q41s+gmVa/q7tXmBB9E9LgVEdex71ISTIxkWDmCg7WlhoUFEUlKjrIQQRujoCzUoLe7Pd7t0ULskKiCqrtXmxN8ENGjDsem4NsDsQCAOS+1R2dO5iE5LCyISDIEQcCnu2Jw4+F85Atf6wS5nOMq6ipO8EFExeLv5eDDX85BEIA3ujTFm12bih0SlUGnwuLChQtPfD02NrZKwRBR3bb17G3sPHcHRnIZvnvDC7YcV0FEVOflKAsxPiwSWfmF8GpaH5+/xJW1pUqnwqJTp06QyWRl/oJUvJ2zthBRZfybnI1Zv8YAAKb2a40uzTmugoiorhMEAR9vu4DY5GzY11Ng1QhvKIyNxA6LyqFTYREXF1dTcRBRHZZbUIigDVHIV2nQo5UdJvZyEzskqgT2ahNRdVv11w38fjGxaLD2W53hYM3B2lKmU2FRkwsbEVHdNXv3JVxNeYBG9RRYNJzjKvQVe7WJqDr99W8qvtl/BQAw+8V28GnGnmyp06mw+Oabb/D+++/D3NwcAPD333/Dx8dHOwd4dnY2PvnkE6xYsaL6IyUig7Q98ja2Rt6GXAYsed0Ldla1Mx85VT/2ahNRdbl5PwcfPBysPdzHBW9xsLZe0KmwCA4OxujRo7WFxcCBAxEdHY0WLYqmg8zNzcXq1atZWBBRhVxLycanu4rGVUzu2xq+bg1Fjoiqgr3aRFQdcguKBmtn5qnQyaU+vvBvx95OPaHT+uePd28/aRpAIqInyStQI2jDOeSp1OjesiGC+rQUOySqRkePHsWIESPg6+uLO3fuAADCwsJw7NgxkSMjIikrHqx9JSkbdlYKrBzRmYO19YhOhQURUXX5/NdLiE0uajgWD/eCEcdVGIzt27fDz88P5ubmOHfuHJRKJQAgMzMT8+bNEzk6IpKy74/ewJ4LiTCWy7ByRGc0tjEXOyTSAQsLIqp1O6JuY/PZW5DJgO9e7wT7ehxXYUj+97//YdWqVfj+++9hYmKi3d69e3dERUWJGBkRSdmxq/fw1R/Fg7U98AwHa+sdnVfe/uGHH2BlZQUAKCwsRGhoKOzs7AAUDd4mInqSaynZ+L+dReMqPnyhFbq1tBM5IqpusbGx6NmzZ6ntNjY2yMjIqP2AiEjybqXlYtIvUdAIwGs+zhjxLMds6SOdCoumTZvi+++/1z53dHREWFhYqX2IiMry6LiKbm4N8f7zrcQOiWqAo6Mjrl27hmbNmpXYfuzYMe1kH0RExfIK1Hg3LBIZuSp4Otvgi5fbc7C2ntKpsIiPj6+hMIioLpj9a8x/4ype78RxFQbqnXfewYcffoi1a9dCJpPh7t27OHHiBKZNm4ZZs2aJHR4RSYggCPhk+wX8k5gFOytTrBrpDTMTDtbWVzoVFvn5+Th48CCGDBkCoGj62eJBeQBgbGyML774AmZmXBWRiEraHnkbW84WrVfx3eud0KgevycM1YwZM6DRaPDCCy8gNzcXPXv2hEKhwPTp0zFu3DixwyMiCfnxWBx+PX8XxnIZlr/Jwdr6TqfB26GhoVi9erX2+bJly3D8+HGcO3cO586dQ1hYmE5rWBw5cgQvvvgimjRpAplMhl27dj31mIiICHTu3BkKhQItW7ZEaGioLikQkQiuJv+3XsWHL7TmuAoDJ5PJ8H//939IS0tDTEwMTp48idTUVNjY2KB58+Zih0dEEnH82j3M2/sPAODTwW3RtQXXMtJ3OhUWGzZswLvvvlti28aNG3H48GEcPnwY8+fPx9atWyt8vpycHHh6emL58uUV2j8uLg6DBw9Gnz59EB0djcmTJ2PcuHHYv3+/LmkQUS3KLSjEexuikKdS47mWdpj0PNerMFRKpRLBwcHw8fFB9+7dsXfvXnh4eODSpUtwd3fHkiVLMGXKFLHDJCIJuJWWi6CNRYO1h3Z2RmC3ZmKHRNVAp1uhrl27hg4dOmifm5mZQS7/rzbp0qULgoKCKny+gQMHYuDAgRXef9WqVWjevDkWLFgAAGjbti2OHTuGRYsWwc/Pr8LnIaLaIQgCPt0Vg6spD2BfT4FFwzmuwpDNmjULq1evRt++fXH8+HEEBARgzJgxOHnyJBYsWICAgAAYGfHeaaK6Lq9AjfFhkUjPVaGjsw2+fIWDtQ2FToVFRkZGiTEVqampJV7XaDQlXq9uJ06cQN++fUts8/Pzw+TJk2vsmkRUeVvP3saOqDuQy4Clb3hxvQoDt3XrVqxfvx4vvfQSYmJi0LFjRxQWFuL8+fP8o4GIABT94DRz50VcTsxCQ0tTrBrBwdqGRKfCwtnZGTExMXB3dy/z9QsXLsDZ2blaAitLUlISHBwcSmxzcHBAVlYW8vLyYG5eesCPUqksUexkZWUBAFQqFVQqlU7XL95f1+OkxhDyYA7SUV4eV5Ky8dnuonEVU15oCW8Xa8nmauifhS7HVsXt27fh7e0NAGjfvj0UCgWmTJnCooKItNb+HY+d5+7ASC7Dsjc7o0l9DtY2JDoVFoMGDcKsWbMwePDgUjM/5eXlYc6cORg8eHC1BlhVISEhmDNnTqntBw4cgIWFRaXOGR4eXtWwJMEQ8mAO0vFoHvlqYMEFIygLZWhbXwPnB1ewd+8VEaOrGEP8LCoqNze3ytdVq9UwNTXVPjc2NtYuqEpEdPx6ycHavm4crG1odCosZs6ciS1btsDd3R2TJk1C69atARStsrps2TIUFhZi5syZNRIoULToUnJycoltycnJsLa2LrO3AiiaEnfq1Kna51lZWXBxcUH//v1hbW2t0/VVKhXCw8PRr18/mJiY6J6ARBhCHsxBOh7PQxAETN5yASn5yXC0VuCnib5oYGH69BOJyFA/C10U9+ZWhSAIGD16NBSKolve8vPzMWHCBFhaWpbYb8eOHVW+FhHplzsZeZi08RzUGgGvejlhNAdrGySdCgsHBwccP34cEydOxIwZMyAIAoCiqQX79euHFStWlLpVqTr5+vpi7969JbaFh4fD19e33GMUCoW2kXuUiYlJpf+AqMqxUmIIeTAH6SjOI/TvOOyNSYaxXIYVI7zRyMby6QdLhKF9FroeU1WBgYElno8YMaJK5zty5Ajmz5+PyMhIJCYmYufOnfD393/iMREREZg6dSouXboEFxcXfPrppxg9enSV4iCiqslXqTEhLBJpOQVo72SNea924C2SBkqnwgIAmjdvjn379iEtLQ3Xrl0DALRs2RK2trY6X/zBgwfacwBF08lGR0fD1tYWTZs2RXBwMO7cuYP169cDACZMmIBly5bh448/xtixY/Hnn39iy5Yt+P3333W+NhFVv6iEdHz5sJt75qC26Ny0gcgRUW1at25dtZ6veErysWPH4tVXX33q/sVTkk+YMAEbNmzAoUOHMG7cODRu3JgzBxKJRBCAWb9exsU7mbDlYG2Dp3NhUczW1hZdunSp0sXPnj2LPn36aJ8X37IUGBiI0NBQJCYmIiEhQft68+bN8fvvv2PKlClYsmQJnJ2d8cMPP7DBIJKAtJwCTNoQBZVawKAOjhjTvZnYIZGe45TkRPrvaJIMO+MTHw7W9oJzg8qNbyX9UOnCojr07t1beztVWcpaVbt37944d+5cDUZFRLrSCMC0bRdxNzMfze0s8fXQjuzmplpXmSnJOXNgScxBOgwhjxPXUrEzvmi9s0/8WuOZpjZ6mY8hfBa1NWugqIUFERmG/bflOHb7PsxM5Fg5ojPqmen/OAXSP5WZkpwzB5aNOUiHvuaRrgS+vWAEDWTwttOgUfol7N17SeywqkRfP4tH1fSsgSwsiKhKjly9h/23i3onQl7tgDaOus22RiQmzhxYEnOQDn3OQ6lS480fz+BBYRacLASseac3rC3Mnn6gROnzZ1GstmYNZGFBRJV2Oz0X07ZehAAZ3uzijFe8am6BTKKnqcyU5Jw5sGzMQTr0LQ9BEDBz12VcuJOF+uYmeNs9D9YWZnqVQ3n07bMoS03PGijXNSAiIqBo+sCJP0chI0+FppYCZg5sI3ZIVMf5+vri0KFDJbY9bUpyIqpeP5+8ia2RtyGXAYuHd0RD/e2ooEpgYUFEOhMEAbN2x+DinUw0sDDBGHc1FMb8OqHq9eDBA0RHRyM6OhrAf1OSF88WGBwcjFGjRmn3nzBhAm7cuIGPP/4YV65cwYoVK7BlyxZMmTJFjPCJ6pzTcWmY89tlAMAnA9qgO1fWrnP4lwAR6WzTmVvYcvbhL1KvdYRt6TtJiKrs7Nmz8PLygpeXF4CiKcm9vLwwa9YsACh3SvLw8HB4enpiwYIFnJKcqJYkZubhvQ2RKNQIeNGzCd7t2ULskEgEHGNBRDo5l5CO2buLZvb4yM8d3dwaYm+syEGRQeKU5ET6IV+lxoSfo3DvQQHaONbD10O5snZdxR4LIqqwlOx8TPw5CgVqDfzaOWBiLzexQyIiIhEJgoDZuy/h/K0M2JibYM1IH1iY8nfruoqFBRFVSEGhBkEbopCUlQ83e0t8G+DJX6SIiOq4DacSsPnsLchlwNI3vNC0IVfWrstYWBBRhXz5+2WciU+HlcIYa0b5cBE8IqI67mx8Gub8VnRr7McD2qBna3uRIyKxsbAgoqfacvYWfjpxEwCwaHgnuNlbiRwRERGJKSkzHxN+joJKLWBwh8YYz8HaBBYWRPQUUQnp+HRnDADgwxdaoZ+Hg8gRERGRmJSFakzcEIl7D5Rwd6iHb4Z15K2xBICFBRE9QXJWPiaERaJArUF/Dwd8+EIrsUMiIiKRff7rJZxLyIC1mTFWj/SGpYKDtakICwsiKlO+So13wyKRkq1EawcrLBzeCXI5f5EiIqrLNp5KwC+nb0EmA757wwvN7CzFDokkhIUFEZUiCAKCd1zUTh/4/SgfWPEXKSKiOi3yZjpm/1p0a+xH/d3R272RyBGR1LCwIKJSVv51HTvP3YGRXIYVb3WGa0P+IkVEVJclZ+Vj4s+RUKkFDGzviPd6cx0jKo2FBRGVcOBSEubvL1pK+/MXPdC9pZ3IERERkZgKCjWY+HPRrbGtGllhPtcxonKwsCAirct3szB5czQEARjxbFOM9G0mdkhERCSyOb9dQlRCBuqZFa1jxFtjqTwsLIgIQFE399s/nUFugRrd3Bpi9ovtxA6JiIhEtul0AjacSigarP26F5pzsDY9gSQKi+XLl6NZs2YwMzND165dcfr06XL3DQ0NhUwmK/EwMzOrxWiJDE9uQSHG/XQWiZn5cLO3xMq3vGFiJImvByIiEklUQjpm7S5aWXtq39bo04aDtenJRP/LYfPmzZg6dSpmz56NqKgoeHp6ws/PDykpKeUeY21tjcTERO3j5s2btRgxkWHRaARM2RyNi3cyYWtpirWjn4GNhYnYYRERkYhSsosGaxeoNfBr54CgPi3FDon0gOiFxcKFC/HOO+9gzJgx8PDwwKpVq2BhYYG1a9eWe4xMJoOjo6P24eDAlYCJKuvLvf9g/6VkmBrJsWakN2eAIiKq4woKNQjaEIXkLCVaNrLCgte4jhFVjKijbwoKChAZGYng4GDtNrlcjr59++LEiRPlHvfgwQO4urpCo9Ggc+fOmDdvHtq1K/t+cKVSCaVSqX2elZUFAFCpVFCpVDrFW7y/rsdJjSHkwRyqR+iJm/jxWBwA4KtX28HTqV6d/HdhCDkAVctD33Mnouozd89lnIlPRz2FMdaM9OZgbaowUf9LuXfvHtRqdakeBwcHB1y5cqXMY9zd3bF27Vp07NgRmZmZ+Pbbb9GtWzdcunQJzs7OpfYPCQnBnDlzSm0/cOAALCwsKhV3eHh4pY6TGkPIgzlU3vn7Mqz7Vw5AhpeaqmF0+xz23j5X6fPxs5COyuSRm5tbA5EQkb7ZcuYWwk4W3WK+aHgntLC3Ejki0id6V4L6+vrC19dX+7xbt25o27YtVq9ejblz55baPzg4GFOnTtU+z8rKgouLC/r37w9ra2udrq1SqRAeHo5+/frBxER/70E3hDyYQ9WcvZmODaGREKDBm12c8fmQtpWek5yfhXRUJY/i3lwiqruib2Xg011FK2tP6dsafT14qznpRtTCws7ODkZGRkhOTi6xPTk5GY6OjhU6h4mJCby8vHDt2rUyX1coFFAoFGUeV9k/IKpyrJQYQh7MQXexSdkY//M5KAs16Nu2Eb54uQOMq2EGKH4W0lGZPAwhbyKqvNRsJSaEFQ3W7ufhgPef52Bt0p2og7dNTU3h7e2NQ4cOabdpNBocOnSoRK/Ek6jValy8eBGNGzeuqTCJDMbt9FyMWnsKWfmF8HZtgKVvdK6WooKIiPSXSq1B0MYoJGXlo4W9JRa+5snB2lQpov9FMXXqVHz//ff46aef8M8//2DixInIycnBmDFjAACjRo0qMbj7iy++wIEDB3Djxg1ERUVhxIgRuHnzJsaNGydWCkR64f4DJUatPY3kLCVaNbLCj4E+MDc1EjssoifiOkdENe/L3//B6bg0WCmMsWakD+qZsQeTKkf0MRbDhw9HamoqZs2ahaSkJHTq1An79u3TDuhOSEiAXP5f/ZOeno533nkHSUlJaNCgAby9vXH8+HF4eHiIlQKR5GXlqzBq7WncSM1BExszrH+7C+pbmIodFtETFa9ztGrVKnTt2hWLFy+Gn58fYmNj0ahR2Qt1WVtbIzY2Vvu8smOHiOqKbZG3EXo8HkDRYO2WjThYmypP9MICACZNmoRJkyaV+VpERESJ54sWLcKiRYtqISoiw5BXoMbboWdw6W4WGlqaImxcVzS2MRc7LKKnenSdIwBYtWoVfv/9d6xduxYzZswo85jidY6I6Oku3s7EzJ0XAQAfvtAK/ThYm6pIEoUFEdUMZaEa43+OLJqP3MwY69/uAjdOHUh6oDbWOQK41tHjmIN01HQe93MK8G7YWRQUavC8uz3e69ms2q/Fz0I6amudIxYWRAaqoFCD936OwpF/U2FuYoTQMc+gXRMbscMiqpDaWOcI4FpH5WEO0lETeag1wIp/5EjMkqORmYD+1onYty+x2q9TjJ+FdNT0OkcsLIgMkEqtwaSNUTh0JQUKYzl+DPSBt6ut2GER1Shd1zkCuNbR45iDdNRkHl/uvYJrWQmwNDXCT+90rbFxFfwspKO21jliYUFkYFRqDT7cdA4HLifD1FiO70f5oFtLO7HDItJJbaxzBHCto/IwB+mo7jx2nruN0BMJAIAFr3VCW6cG1Xbu8vCzkI6aXudI9Olmiaj6FBQW9VTsvZgEUyM5Vo/0Rs/W9mKHRaQzrnNEVP1i7mRixvaiwdqT+rTEgPac6ICqF3ssiAxEvkqN9zZE4c8rKTA1lmPViM7o4172lJxE+mDq1KkIDAyEj48PunTpgsWLF5da58jJyQkhISEAitY5evbZZ9GyZUtkZGRg/vz5XOeI6KG0nAKMD4uEslCDPu72mNKvtdghkQFiYUFkAHILCjE+LBJHr96DmYkca0b6sKeC9B7XOSKqHoUPx93dychDs4YWWPy6F4y4sjbVABYWRHouI7cAY0PPICohAxamRvgx8Bn4ujUUOyyiasF1joiq7qs/ruD49fuwMDXCmlE+sDHX73ECJF0sLIj0WHJWPkb9eBqxydmwNjPGujHPcPYnIiLS2h19Bz8ciwMAfBvgidYO9USOiAwZCwsiPXU99QFGrzuNW2l5aFRPgbC3u8LdkQ0GEREVuXQ3E59svwAAeK+3GwZ14EQGVLNYWBDpoTPxaXhn/Vlk5Krg2tACP7/dFS62lVvMi4iIDE/6w8Ha+SoNerW2x7T+7mKHRHUACwsiPbPnwl1M3XIeBYUadHKpjx8CfWBnVXoefiIiqpsK1Rq8/8s53E7PQ1NbC3zHwdpUS1hYEOkJjUbAkkNXseTQVQCAXzsHLB7uBXNTI5EjIyIiKZm/PxbHrt2DuYkR1ozyho0FB2tT7WBhQaQHcpSFmLblPPZdSgIAjO3eHP83uC1/gSIiohJ+PX8Xq4/cAADMD+iINo7WIkdEdQkLCyKJi7+Xgwk/R+JKUjZMjGT40r8DXnvGReywiIhIYv5JzMLH284DACb0csOQjk1EjojqGhYWRBK2LyYR07deQLayEHZWCqwe2ZnTyRIRUSkZuQV4N+ws8lUa9Ghlh+l+HKxNtY+FBZEEKQvV+GZfLH58OPf4M80aYOkbneFoYyZyZEREJDVqjYD3fzmHW2l5cLE152BtEg0LCyKJuZaSjQ9+icblxCwAwLs9W2C6nztMjOQiR0ZERFI0f38sjl59OFh7pA8aWJqKHRLVUZL4S2X58uVo1qwZzMzM0LVrV5w+ffqJ+2/duhVt2rSBmZkZOnTogL1799ZSpEQ1R6MRsP5EPAZ/dwyXE7PQwMIEa0Z6Y+agtiwqiIioTL9fSMSqv64DAL4e1hFtG3OwNolH9L9WNm/ejKlTp2L27NmIioqCp6cn/Pz8kJKSUub+x48fxxtvvIG3334b586dg7+/P/z9/RETE1PLkRNVn/h7OXjj+5OYtfsSlIVF98fun9wT/ds5ih0aERFJ1JWkLHy0tWiw9rs9W+AlTw7WJnGJXlgsXLgQ77zzDsaMGQMPDw+sWrUKFhYWWLt2bZn7L1myBAMGDMD06dPRtm1bzJ07F507d8ayZctqOXKiqlNrgB+OxWPAkiM4FZcGcxMjzH7RAz+N6YJG1hxPQUREZcvMVWF8WCTyVGo819IOH3OwNkmAqGMsCgoKEBkZieDgYO02uVyOvn374sSJE2Uec+LECUydOrXENj8/P+zatavM/ZVKJZRKpfZ5VlbRfesqlQoqlUqneLdH3sLFFBnyo25BYWICI7kMxnIZjI1kMJLLYGokh7FcBhMj+cOHDCbGcpgayWFqLIfi4cNYLoNMJt6gquK8dc1fSgwhh6P/puCbC0ZIyvsXANCthS3mvuyBprYWUKsLoVaLHGAFGcJnYQg5AFXLQ99zJ6pL1BoBH2w6h5v3c+HcwBxL3/CCMW+ZJQkQtbC4d+8e1Go1HBwcSmx3cHDAlStXyjwmKSmpzP2TkpLK3D8kJARz5swptf3AgQOwsLDQKd45p42QpzbChuv/6HTc42QQYCKH9mEqB0yNHv6vXIDCCEUPOaAwBsyMBJgZAWZGgLkRYG4swNwIsDAGzI2LjqtMnRIeHl6lPKRAH3NIzQP23JIj+r4cgAyWxgJectWgq30KYk6mQF9v6tPHz+JxhpADULk8cnNzayASIqoJC8Nj8de/qTAzkWP1SG8O1ibJMPhZoYKDg0v0cGRlZcHFxQX9+/eHtbVuA5z2Zp5Dwt1k1G/QEAKAQo2AQo0AtUaASi2gUK2BSi1ApdagUFP0vwWFGhQ83F5MgAwFGqBAU9ZVdK8QTI3lqG9ugvrmJmhgaQJbC1PYWpqioaUpbK1MYWdpCvt6CthZmaJRPQWMoEF4eDj69esHExMTna8nBSqVSu9yuPdAiWWHb2Dzhdso1AiQy4DuDhp8M7In7Kx1K3KlRB8/i8cZQg5A1fIo7s0lImn742Iilh9+OFh7aEe0a2IjckRE/xG1sLCzs4ORkRGSk5NLbE9OToajY9mDVh0dHXXaX6FQQKFQlNpuYmKic8O77A0v7N27F4MGPaPzsRqNgAK1BkqVBspCNfJVGuQXqpGvUiOvQI1clRr5BWrkFKiRV1CInAI1cpSFeKAsRI6yENn5xQ8VsvMLkZmnQmaeCoUaAQWFGqRkK5GSrXx6IACszYxhITPC1tQLaFLfHI425mhiY4Ym9c3RpL45nOqbw9zUSKf8xFKZz7G2JWbm4fsjcfjldALyVEX3N/VqbY9pfVsi7txR2FlbSD6HitCHz+JpDCEHoHJ5GELeRIbu3+RsTHs4WHvcc83xcicnkSMiKknUwsLU1BTe3t44dOgQ/P39AQAajQaHDh3CpEmTyjzG19cXhw4dwuTJk7XbwsPD4evrWwsRV55cLoOZ3AhmJkYAqqcBFwQBuQVqpOcWICNXhfTcAqTl/Pe496AA9x4ocf+BEqkPlEjJUkJZqEFWfiGyIEPStfvlntvOyhRODSzg3MAcTW0t4NLAAk1tLeDa0AJN6ptz4Z0K+CcxC6F/x2PHudvaHqtOLvXxyYA28HVrCJVKhbhzIgdJRER6ITNPhXfXn0VugRrd3BpixsA2YodEVIrot0JNnToVgYGB8PHxQZcuXbB48WLk5ORgzJgxAIBRo0bByckJISEhAIAPP/wQvXr1woIFCzB48GBs2rQJZ8+exZo1a8RMQxQymQyWCmNYKozh3ODp+wuCgGxlIe7cf4DfDh6Fa9uOSH2gwt3MfCRl5uNuRh7upOchW1n4sCgpwPlbGaXOY2Ikg0uDoiKjmZ0lmj/yaGJjDnkdLjryVWqEX05G2MmbOB2Xpt3etbktJj3fEs+1tBN14D4REekftUbA5E3nEH8/F071zbHszc4crE2SJHphMXz4cKSmpmLWrFlISkpCp06dsG/fPu0A7YSEBMjl//3j6datGzZu3IhPP/0UM2fORKtWrbBr1y60b99erBT0hkwmg7WZCcwbWcG9voBBXk5l3v6QmafC7fRc3ErLe/i/ubiZlouEtFzcTstDgVqDG/dycONeDhCbWuJYU2M5mje0RAv7hw87K7g1skILe0tYmxnmrRZqjYCohHTsPHcHe87fRVZ+IQDASC7DgHaOGPtcM3i72oocJRER6avFB//F4dhUKIyLBmvbcrA2SZTohQUATJo0qdxbnyIiIkptCwgIQEBAQA1HVXfZmJvAxtymzAFhao2ApKx83LyXg7j7OYi/l4O4e7mIu/cACWm5KCjUIDY5G7HJ2aWOtbNSoIW9JdweFhzN7YqKDxdbC71bWTpHWYhTcfcRfjkZ4ZdTcO/Bf+NbGtuYYZi3M97q6gpHG65FQVQVy5cvx/z585GUlARPT08sXboUXbp0KXf/rVu34rPPPkN8fDxatWqFr7/+GoMGDarFiImq14HLyVj65zUAwFdDO6C9Ewdrk3RJorAg/WEkl8Hp4QDvbi3tSrxWqNbgTkYebqTm4Hrqg6JejdQHuJGag5RsJe49KHo8eotQ8TldGpijmZ0lmjW0hGvDotusmtpawrmB+cNxKeJKyylA9K10nEvIwMkb93EuIQOFmv9m+qpnZox+Hg4Y1tkZz7ZoWKdvByOqLps3b8bUqVOxatUqdO3aFYsXL4afnx9iY2PRqFGjUvsfP34cb7zxBkJCQjBkyBBs3LgR/v7+iIqKYq826aU7OcDy7UWTkI/t3hyveDmLHBHRk7GwoGpjbCSHa0NLuDa0RJ82JRv97HwV4u7l4Ebqw2Lj4f+Pu5eDPJUa8fdzEX8/F0BqqfM2qqeAU4OiYqZJfXM4WpvBztIYN7KAm/dz4djAEpamRlUeu6BSa5CUmY9b6bm4nZ6H66kPcDX5Af5Nzsbt9LxS+ze1tUDP1nbwa+eIrs0bwtRYv3pdiKRu4cKFeOedd7Rj7latWoXff/8da9euxYwZM0rtv2TJEgwYMADTp08HAMydOxfh4eFYtmwZVq1aVauxE1WFslCN5X9ex/KLRlALajzbwhYzB3GwNkkfCwuqFfXMTNDRuT46OtcvsV0QBCRnKXHj3gPcvJ+L+Ie3VyWk5SHhfg5yCtTaqXTPJWQ8dlZjLLl0DEDR2A6bh2t51DMzhoWpMSxMjaAwMYKxXKadxUqjEaAWBOSr1Mh9OKVvRp4K9x8UIDPvySsPu9lbopNLA/g0a4DubnZo2lB/154gkrqCggJERkYiODhYu00ul6Nv3744ceJEmcecOHGixLpFAODn54ddu3aVex2lUgml8r9bGYvX81CpVDqtRn7s2n3suXAXd+7IcWTHxRJjA/WJRqNhDhIQeTMdN+7lApDhOTdbLAjoCEGjhkqjFjs0nRT/G9Ll35IUGUIeVclBl2NYWJCoZDIZHG3M4Ghjhm5uJV8TBAFpOQW483C2qjsZebibkY/krHwkZubhZnI6cjVGyFMVLUSYmq1EagXX8iiPqbEczvXN4dTAHM0aWqK1gxVaOdRDW0dr2FgY5uBzIim6d+8e1Gq1diKPYg4ODrhy5UqZxyQlJZW5f1JSUrnXCQkJwZw5c0ptP3DgACwsKv7jQUSiDDvjjQDIgZTECh8nTcxBCuqZCHi1mQZeDVNw8q+DYodTJeHh4WKHUC0MIY/K5JCbm1vhfVlYkGTJZDI0tFKgoZWiVE+HSqV6uFihHwo0MqTnFvU4ZOaqkK0sRF6BGjkFhSgo1GhXRgcAIzkgl8lgZmIES4URLEyNYW1mAvt6pmhoqYCNuQnHRxDVIcHBwSV6ObKysuDi4oL+/fvD2tq6wudxvp0J16upuHbtKlq2bAUjPf2lXK3RMAcJsFQYY6CHHU4fi0C/fv30dgFLlUqF8PBwvc4BMIw8qpJDcU9uRbCwIL2ny1oeRKQf7OzsYGRkhOTk5BLbk5OT4ejoWOYxjo6OOu0PAAqFAgqFotR2XVcv925uh47ONtib9y8G9Wmp1398MAdpKL79RNf/FqXIEHIADCOPyuSgy/76WcoTEZFBMzU1hbe3Nw4dOqTdptFocOjQIfj6+pZ5jK+vb4n9gaJu//L2JyKi6sUeCyIikqSpU6ciMDAQPj4+6NKlCxYvXoycnBztLFGjRo2Ck5MTQkJCAAAffvghevXqhQULFmDw4MHYtGkTzp49izVr1oiZBhFRncHCgoiIJGn48OFITU3FrFmzkJSUhE6dOmHfvn3aAdoJCQklZv3p1q0bNm7ciE8//RQzZ85Eq1atsGvXLq5hQURUS1hYEBGRZE2aNAmTJk0q87WIiIhS2wICAhAQEFDDURERUVk4xoKIiIiIiKqMhQUREREREVVZnbsVShCK1jPQZU7eYiqVCrm5ucjKytLr6cYMIQ/mIB2GkIch5ABULY/i78Ti78i6qq63EcxBOgwhD0PIATCMPGqrfahzhUV2djYAwMXFReRIiIikJzs7GzY2NmKHIRq2EUREZatI+yAT6tjPUxqNBnfv3kW9evUgk+m2wnLxiqy3bt3SaUVWqTGEPJiDdBhCHoaQA1C1PARBQHZ2Npo0aVJipqW6pq63EcxBOgwhD0PIATCMPGqrfahzPRZyuRzOzs5VOoe1tbXe/of1KEPIgzlIhyHkYQg5AJXPoy73VBRjG1GEOUiHIeRhCDkAhpFHTbcPdfdnKSIiIiIiqjYsLIiIiIiIqMpYWOhAoVBg9uzZUCgUYodSJYaQB3OQDkPIwxByAAwnD31lCO8/c5AOQ8jDEHIADCOP2sqhzg3eJiIiIiKi6sceCyIiIiIiqjIWFkREREREVGUsLIiIiIiIqMpYWFTSSy+9hKZNm8LMzAyNGzfGyJEjcffuXbHD0kl8fDzefvttNG/eHObm5nBzc8Ps2bNRUFAgdmg6+fLLL9GtWzdYWFigfv36YodTYcuXL0ezZs1gZmaGrl274vTp02KHpJMjR47gxRdfRJMmTSCTybBr1y6xQ9JZSEgInnnmGdSrVw+NGjWCv78/YmNjxQ5LJytXrkTHjh21c5P7+vrijz/+EDusOk/f2whDaR8A/Wwj2D6IzxDaB6D22wgWFpXUp08fbNmyBbGxsdi+fTuuX7+OYcOGiR2WTq5cuQKNRoPVq1fj0qVLWLRoEVatWoWZM2eKHZpOCgoKEBAQgIkTJ4odSoVt3rwZU6dOxezZsxEVFQVPT0/4+fkhJSVF7NAqLCcnB56enli+fLnYoVTaX3/9haCgIJw8eRLh4eFQqVTo378/cnJyxA6twpydnfHVV18hMjISZ8+exfPPP4+XX34Zly5dEju0Ok3f2whDaR8A/Wsj2D5IgyG0D4AIbYRA1WL37t2CTCYTCgoKxA6lSr755huhefPmYodRKevWrRNsbGzEDqNCunTpIgQFBWmfq9VqoUmTJkJISIiIUVUeAGHnzp1ih1FlKSkpAgDhr7/+EjuUKmnQoIHwww8/iB0GPcIQ2gh9bh8EQX/aCLYP0mQo7YMg1GwbwR6LapCWloYNGzagW7duMDExETucKsnMzIStra3YYRi0goICREZGom/fvtptcrkcffv2xYkTJ0SMjDIzMwFAb/8NqNVqbNq0CTk5OfD19RU7HHrIUNoItg81j+2DdOl7+wDUThvBwqIKPvnkE1haWqJhw4ZISEjA7t27xQ6pSq5du4alS5di/PjxYodi0O7duwe1Wg0HB4cS2x0cHJCUlCRSVKTRaDB58mR0794d7du3FzscnVy8eBFWVlZQKBSYMGECdu7cCQ8PD7HDqvMMqY1g+1A72D5Ikz63D0DtthEsLB4xY8YMyGSyJz6uXLmi3X/69Ok4d+4cDhw4ACMjI4waNQqCBNYb1DUPALhz5w4GDBiAgIAAvPPOOyJF/p/K5EBUFUFBQYiJicGmTZvEDkVn7u7uiI6OxqlTpzBx4kQEBgbi8uXLYodlcAyhjTCE9gFgG0G1S5/bB6B22wiuvP2I1NRU3L9//4n7tGjRAqampqW23759Gy4uLjh+/LjotyDomsfdu3fRu3dvPPvsswgNDYVcLn69WZnPIjQ0FJMnT0ZGRkYNR1c1BQUFsLCwwLZt2+Dv76/dHhgYiIyMDL38VVMmk2Hnzp0l8tEnkyZNwu7du3HkyBE0b95c7HCqrG/fvnBzc8Pq1avFDsWgGEIbYQjtA2C4bQTbB+kxtPYBqNk2wrjaz6jH7O3tYW9vX6ljNRoNAECpVFZnSJWiSx537txBnz594O3tjXXr1kmm0ajKZyF1pqam8Pb2xqFDh7RftBqNBocOHcKkSZPEDa6OEQQB77//Pnbu3ImIiAiDaTQ0Go0kvosMjSG0EYbQPgCG20awfZAOQ20fgJptI1hYVMKpU6dw5swZPPfcc2jQoAGuX7+Ozz77DG5ubqL3Vujizp076N27N1xdXfHtt98iNTVV+5qjo6OIkekmISEBaWlpSEhIgFqtRnR0NACgZcuWsLKyEje4ckydOhWBgYHw8fFBly5dsHjxYuTk5GDMmDFih1ZhDx48wLVr17TP4+LiEB0dDVtbWzRt2lTEyCouKCgIGzduxO7du1GvXj3tPcw2NjYwNzcXObqKCQ4OxsCBA9G0aVNkZ2dj48aNiIiIwP79+8UOrc4yhDbCUNoHQP/aCLYP0mAI7QMgQhtRI3NNGbgLFy4Iffr0EWxtbQWFQiE0a9ZMmDBhgnD79m2xQ9PJunXrBABlPvRJYGBgmTkcPnxY7NCeaOnSpULTpk0FU1NToUuXLsLJkyfFDkknhw8fLvN9DwwMFDu0Civvv/9169aJHVqFjR07VnB1dRVMTU0Fe3t74YUXXhAOHDggdlh1miG0EYbSPgiCfrYRbB/EZwjtgyDUfhvBMRZERERERFRl0rlhkoiIiIiI9BYLCyIiIiIiqjIWFkREREREVGUsLIiIiIiIqMpYWBARERERUZWxsCAiIiIioipjYUFERERERFXGwoKIiIiIiKqMhQUREREREVUZCwsiIiIiIqoyFhZERERERFRlLCyIallqaiocHR0xb9487bbjx4/D1NQUhw4dEjEyIiISE9sH0ncyQRAEsYMgqmv27t0Lf39/HD9+HO7u7ujUqRNefvllLFy4UOzQiIhIRGwfSJ+xsCASSVBQEA4ePAgfHx9cvHgRZ86cgUKhEDssIiISGdsH0lcsLIhEkpeXh/bt2+PWrVuIjIxEhw4dxA6JiIgkgO0D6SuOsSASyfXr13H37l1oNBrEx8eLHQ4REUkE2wfSV+yxIBJBQUEBunTpgk6dOsHd3R2LFy/GxYsX0ahRI7FDIyIiEbF9IH3GwoJIBNOnT8e2bdtw/vx5WFlZoVevXrCxscGePXvEDo2IiETE9oH0GW+FIqplERERWLx4McLCwmBtbQ25XI6wsDAcPXoUK1euFDs8IiISCdsH0nfssSAiIiIioipjjwUREREREVUZCwsiIiIiIqoyFhZERERERFRlLCyIiIiIiKjKWFgQEREREVGVsbAgIiIiIqIqY2FBRERERERVxsKCiIiIiIiqjIUFERERERFVGQsLIiIiIiKqMhYWRERERERUZSwsiIiIiIioyv4f8bkWo5d7IwsAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 800x300 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "gelu, relu = GELU(), nn.ReLU()\n",
    "\n",
    "# Some sample data\n",
    "x = torch.linspace(-3, 3, 100)\n",
    "y_gelu, y_relu = gelu(x), relu(x)\n",
    "\n",
    "plt.figure(figsize=(8, 3))\n",
    "for i, (y, label) in enumerate(zip([y_gelu, y_relu], [\"GELU\", \"ReLU\"]), 1):\n",
    "    plt.subplot(1, 2, i)\n",
    "    plt.plot(x, y)\n",
    "    plt.title(f\"{label} activation function\")\n",
    "    plt.xlabel(\"x\")\n",
    "    plt.ylabel(f\"{label}(x)\")\n",
    "    plt.grid(True)\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08a918a2",
   "metadata": {},
   "source": [
    "- 如图所示，ReLU是一种分段线性函数：当输入值为正时直接输出该输入值，否则输出零；\n",
    "- GELU则是一种平滑的非线性函数，其形态近似ReLU但对负值具有非零梯度；\n",
    "- 接下来我们将实现小型神经网络模块FeedForward，该模块将用于后续大语言模型中的Transformer块；\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "544bc6e4",
   "metadata": {},
   "source": [
    "![](./images/gelu.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "f9660608",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "768"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "GPT_CONFIG_124M[\"emb_dim\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "0aac4606",
   "metadata": {},
   "outputs": [],
   "source": [
    "class FeedForward(nn.Module):\n",
    "    def __init__(self, cfg):\n",
    "        super().__init__()\n",
    "\n",
    "        self.layers = nn.Sequential(\n",
    "            nn.Linear(cfg[\"emb_dim\"], 4*cfg[\"emb_dim\"]),\n",
    "            GELU(),\n",
    "            nn.Linear(4*cfg[\"emb_dim\"], cfg[\"emb_dim\"])\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        return self.layers(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87d2969f",
   "metadata": {},
   "source": [
    "![](./images/mlp-framework.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "a55d1e62",
   "metadata": {},
   "outputs": [],
   "source": [
    "ffn = FeedForward(GPT_CONFIG_124M)\n",
    "\n",
    "x = torch.rand(2, 3, 768)\n",
    "out = ffn(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "5d969c23",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 3, 768])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "out.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "fd3b209a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Parameter containing:\n",
       "tensor([[-0.0223,  0.0308,  0.0022,  ..., -0.0174, -0.0227, -0.0178],\n",
       "        [-0.0027, -0.0043, -0.0274,  ...,  0.0079,  0.0112,  0.0010],\n",
       "        [-0.0029,  0.0175,  0.0244,  ..., -0.0164, -0.0300,  0.0045],\n",
       "        ...,\n",
       "        [ 0.0255,  0.0023,  0.0277,  ..., -0.0037, -0.0134, -0.0284],\n",
       "        [ 0.0266,  0.0236, -0.0193,  ...,  0.0120,  0.0061, -0.0261],\n",
       "        [ 0.0222,  0.0106, -0.0190,  ..., -0.0253,  0.0320, -0.0109]],\n",
       "       requires_grad=True)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ffn.layers[0].weight\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fac72082",
   "metadata": {},
   "source": [
    "![](./images/scenario-gelu.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aea9b1b9",
   "metadata": {},
   "source": [
    "## 添加快捷连接​​\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "721f92e7",
   "metadata": {},
   "source": [
    "- 接下来我们讨论快捷连接（亦称跳跃连接或残差连接）的核心思想；\n",
    "- 该概念最初为解决计算机视觉领域深度网络中的梯度消失问题而提出（参见残差网络）；\n",
    "- 快捷连接通过构建梯度传播的短路径，为网络创建替代性高速通路；\n",
    "- 其实现方式是将某一层的输出与后续层（通常跨越一个或多个中间层）的输出直接相加；\n",
    "\n",
    "下面通过一个简易示例网络演示该机制："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a040ed40",
   "metadata": {},
   "source": [
    "![](./images/res.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "740a64a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "class ExampleDeepNeuralNetwork(nn.Module):\n",
    "    def __init__(self, layer_sizes, use_shortcut):\n",
    "        super().__init__()\n",
    "        self.use_shortcut = use_shortcut\n",
    "        self.layers = nn.ModuleList([\n",
    "            nn.Sequential(nn.Linear(layer_sizes[0], layer_sizes[1]), GELU()),\n",
    "            nn.Sequential(nn.Linear(layer_sizes[1], layer_sizes[2]), GELU()),\n",
    "            nn.Sequential(nn.Linear(layer_sizes[2], layer_sizes[3]), GELU()),\n",
    "            nn.Sequential(nn.Linear(layer_sizes[3], layer_sizes[4]), GELU()),\n",
    "            nn.Sequential(nn.Linear(layer_sizes[4], layer_sizes[5]), GELU())\n",
    "        ])\n",
    "\n",
    "    def forward(self, x):\n",
    "        for layer in self.layers:\n",
    "            # Compute the output of the current layer\n",
    "            layer_output = layer(x)\n",
    "            # Check if shortcut can be applied\n",
    "            if self.use_shortcut and x.shape == layer_output.shape:\n",
    "                x = x + layer_output\n",
    "            else:\n",
    "                x = layer_output\n",
    "        return x\n",
    "\n",
    "\n",
    "def print_gradients(model, x):\n",
    "    # Forward pass\n",
    "    output = model(x)\n",
    "    target = torch.tensor([[0.]])\n",
    "\n",
    "    # Calculate loss based on how close the target\n",
    "    # and output are\n",
    "    loss = nn.MSELoss()\n",
    "    loss = loss(output, target)\n",
    "    \n",
    "    # Backward pass to calculate the gradients\n",
    "    loss.backward()\n",
    "\n",
    "    for name, param in model.named_parameters():\n",
    "        if 'weight' in name:\n",
    "            # Print the mean absolute gradient of the weights\n",
    "            print(f\"{name} has gradient mean of {param.grad.abs().mean().item()}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "69a1ebc5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "layers.0.0.weight has gradient mean of 0.00020173587836325169\n",
      "layers.1.0.weight has gradient mean of 0.00012011159560643137\n",
      "layers.2.0.weight has gradient mean of 0.0007152040489017963\n",
      "layers.3.0.weight has gradient mean of 0.0013988736318424344\n",
      "layers.4.0.weight has gradient mean of 0.005049645435065031\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/jason/gench/build-LLM-from-scratch/.venv/lib/python3.10/site-packages/torch/nn/modules/loss.py:610: UserWarning: Using a target size (torch.Size([1, 1])) that is different to the input size (torch.Size([1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
      "  return F.mse_loss(input, target, reduction=self.reduction)\n"
     ]
    }
   ],
   "source": [
    "layer_sizes = [3, 3, 3, 3, 3, 1]\n",
    "sample_input = torch.tensor([1., 0., -1.])\n",
    "\n",
    "torch.manual_seed(123)\n",
    "output_without_shortcut = ExampleDeepNeuralNetwork(\n",
    "    layer_sizes=layer_sizes, use_shortcut=False\n",
    ")\n",
    "\n",
    "print_gradients(output_without_shortcut, sample_input)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "8487a03f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "layers.0.0.weight has gradient mean of 0.22169798612594604\n",
      "layers.1.0.weight has gradient mean of 0.20694111287593842\n",
      "layers.2.0.weight has gradient mean of 0.3289700150489807\n",
      "layers.3.0.weight has gradient mean of 0.26657330989837646\n",
      "layers.4.0.weight has gradient mean of 1.3258544206619263\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(123)\n",
    "\n",
    "output_with_shortcut = ExampleDeepNeuralNetwork(\n",
    "    layer_sizes=layer_sizes, use_shortcut=True\n",
    ")\n",
    "\n",
    "print_gradients(output_with_shortcut, sample_input)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb790047",
   "metadata": {},
   "source": [
    "## ​Transformer块中注意力层与线性层的连接整合"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21b7eada",
   "metadata": {},
   "source": [
    "- 本节将整合前述概念，构建所谓的​​Transformer块​​；\n",
    "- 一个Transformer块将上一章的​​因果多头注意力模块​​与​​线性层​​（即前文实现的前馈神经网络）相结合；\n",
    "- 此外，Transformer块还引入了​​Dropout机制​​与​​快捷连接​​以提升训练稳定性；"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "b07cf302",
   "metadata": {},
   "outputs": [],
   "source": [
    "class MultiHeadAttention(nn.Module):\n",
    "    def __init__(self, d_in, d_out, context_length, dropout, num_heads, qkv_bias=False):\n",
    "        super().__init__()\n",
    "        assert d_out % num_heads == 0, \"d_out must be divisible by num_heads\"\n",
    "\n",
    "        self.d_out = d_out\n",
    "        self.num_heads = num_heads\n",
    "        self.head_dim = d_out // num_heads  # Reduce the projection dim to match desired output dim\n",
    "\n",
    "        self.W_query = nn.Linear(d_in, d_out, bias=qkv_bias)\n",
    "        self.W_key = nn.Linear(d_in, d_out, bias=qkv_bias)\n",
    "        self.W_value = nn.Linear(d_in, d_out, bias=qkv_bias)\n",
    "        self.out_proj = nn.Linear(d_out, d_out)  # Linear layer to combine head outputs\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "        self.register_buffer('mask', torch.triu(torch.ones(context_length, context_length), diagonal=1))\n",
    "\n",
    "    def forward(self, x):\n",
    "        b, num_tokens, d_in = x.shape\n",
    "\n",
    "        keys = self.W_key(x)  # Shape: (b, num_tokens, d_out)\n",
    "        queries = self.W_query(x)\n",
    "        values = self.W_value(x)\n",
    "\n",
    "        # We implicitly split the matrix by adding a `num_heads` dimension\n",
    "        # Unroll last dim: (b, num_tokens, d_out) -> (b, num_tokens, num_heads, head_dim)\n",
    "        keys = keys.view(b, num_tokens, self.num_heads, self.head_dim)\n",
    "        values = values.view(b, num_tokens, self.num_heads, self.head_dim)\n",
    "        queries = queries.view(b, num_tokens, self.num_heads, self.head_dim)\n",
    "\n",
    "        # Transpose: (b, num_tokens, num_heads, head_dim) -> (b, num_heads, num_tokens, head_dim)\n",
    "        keys = keys.transpose(1, 2)\n",
    "        queries = queries.transpose(1, 2)\n",
    "        values = values.transpose(1, 2)\n",
    "\n",
    "        # Compute scaled dot-product attention (aka self-attention) with a causal mask\n",
    "        attn_scores = queries @ keys.transpose(2, 3)  # Dot product for each head\n",
    "\n",
    "        # Original mask truncated to the number of tokens and converted to boolean\n",
    "        mask_bool = self.mask.bool()[:num_tokens, :num_tokens]\n",
    "\n",
    "        # Use the mask to fill attention scores\n",
    "        attn_scores.masked_fill_(mask_bool, -torch.inf)\n",
    "\n",
    "        attn_weights = torch.softmax(attn_scores / keys.shape[-1]**0.5, dim=-1)\n",
    "        attn_weights = self.dropout(attn_weights)\n",
    "\n",
    "        # Shape: (b, num_tokens, num_heads, head_dim)\n",
    "        context_vec = (attn_weights @ values).transpose(1, 2)\n",
    "\n",
    "        # Combine heads, where self.d_out = self.num_heads * self.head_dim\n",
    "        context_vec = context_vec.contiguous().view(b, num_tokens, self.d_out)\n",
    "        context_vec = self.out_proj(context_vec)  # optional projection\n",
    "\n",
    "        return context_vec"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae652285",
   "metadata": {},
   "source": [
    "![](./images/transformer.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "e12dd32e",
   "metadata": {},
   "outputs": [],
   "source": [
    "class TransformerBlock(nn.Module):\n",
    "    def __init__(self, cfg):\n",
    "        super().__init__()\n",
    "        self.att = MultiHeadAttention(\n",
    "            d_in=cfg[\"emb_dim\"],\n",
    "            d_out=cfg[\"emb_dim\"],\n",
    "            context_length=cfg[\"context_length\"],\n",
    "            num_heads=cfg[\"n_heads\"], \n",
    "            dropout=cfg[\"drop_rate\"],\n",
    "            qkv_bias=cfg[\"qkv_bias\"])\n",
    "        self.ff = FeedForward(cfg)\n",
    "        self.norm1 = LayerNorm(cfg[\"emb_dim\"])\n",
    "        self.norm2 = LayerNorm(cfg[\"emb_dim\"])\n",
    "        self.drop_shortcut = nn.Dropout(cfg[\"drop_rate\"])\n",
    "\n",
    "    def forward(self, x):\n",
    "        # Shortcut connection for attention block\n",
    "        shortcut = x\n",
    "        x = self.norm1(x)\n",
    "        x = self.att(x)  # Shape [batch_size, num_tokens, emb_size]\n",
    "        x = self.drop_shortcut(x)\n",
    "        x = x + shortcut  # Add the original input back\n",
    "\n",
    "        # Shortcut connection for feed forward block\n",
    "        shortcut = x\n",
    "        x = self.norm2(x)\n",
    "        x = self.ff(x)\n",
    "        x = self.drop_shortcut(x)\n",
    "        x = x + shortcut  # Add the original input back\n",
    "\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9036cd76",
   "metadata": {},
   "source": [
    "- 假设我们拥有2个输入样本，每个样本包含4个词元，且每个词元为768维嵌入向量；\n",
    "- 该Transformer块将依次应用自注意力机制与线性层进行处理，最终生成相同维度的输出；\n",
    "- 可以将其理解为：输出是对前一章所讨论上下文向量的增强版本；\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "b23bbff3",
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.manual_seed(123)\n",
    "\n",
    "x = torch.rand(2, 4, 768)\n",
    "block = TransformerBlock(GPT_CONFIG_124M)\n",
    "\n",
    "output = block(x)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "7728e20d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 4, 768])"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "7b08ebfe",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 4, 768])"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "895edcd7",
   "metadata": {},
   "source": [
    "![](./images/scenario-transformer.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e014bb46",
   "metadata": {},
   "source": [
    "## 实现GPT大语言模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e1bd993",
   "metadata": {},
   "source": [
    "- 我们即将完成整体架构：现在将本章开头编码实现的Transformer块集成至主体框架中，从而构建出可用的GPT模型架构；\n",
    "- 需注意，Transformer块在模型中会重复堆叠多次：以最小的1.24亿参数GPT-2模型为例，其堆叠层数为12次；"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc83c4aa",
   "metadata": {},
   "source": [
    "![](./images/gpt.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "fea7bbcf",
   "metadata": {},
   "outputs": [],
   "source": [
    "class GPTModel(nn.Module):\n",
    "    def __init__(self, cfg):\n",
    "        super().__init__()\n",
    "        self.tok_emb = nn.Embedding(cfg[\"vocab_size\"], cfg[\"emb_dim\"])\n",
    "        self.pos_emb = nn.Embedding(cfg[\"context_length\"], cfg[\"emb_dim\"])\n",
    "        self.drop_emb = nn.Dropout(cfg[\"drop_rate\"])\n",
    "        \n",
    "        # 占位\n",
    "        self.trf_blocks = nn.Sequential(\n",
    "            *[TransformerBlock(cfg) for _ in range(cfg[\"n_layers\"])])\n",
    "        \n",
    "        # 占位\n",
    "        self.final_norm = LayerNorm(cfg[\"emb_dim\"])\n",
    "        self.out_head = nn.Linear(\n",
    "            cfg[\"emb_dim\"], cfg[\"vocab_size\"], bias=False\n",
    "        )\n",
    "\n",
    "    def forward(self, in_idx):\n",
    "        batch_size, seq_len = in_idx.shape\n",
    "        tok_embeds = self.tok_emb(in_idx)\n",
    "        pos_embeds = self.pos_emb(torch.arange(seq_len, device=in_idx.device))\n",
    "        x = tok_embeds + pos_embeds\n",
    "        x = self.drop_emb(x)\n",
    "        x = self.trf_blocks(x)\n",
    "        x = self.final_norm(x)\n",
    "        logits = self.out_head(x)\n",
    "        return logits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "2033a9a1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[16833,  3626,  6100,   345],\n",
       "        [16833,  1110,   423,  1257]])"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import tiktoken\n",
    "\n",
    "tokenizer = tiktoken.get_encoding(\"gpt2\")\n",
    "\n",
    "batch = []\n",
    "txt_1 = \"every effort moves you\"\n",
    "txt_2 = \"every day have fun\"\n",
    "\n",
    "batch.append(torch.tensor(tokenizer.encode(txt_1)))\n",
    "batch.append(torch.tensor(tokenizer.encode(txt_2)))\n",
    "\n",
    "batch = torch.stack(batch, dim=0)\n",
    "\n",
    "batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "7fb0e49e",
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.manual_seed(123)\n",
    "\n",
    "model = GPTModel(GPT_CONFIG_124M)\n",
    "output = model(batch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "327cc5b7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 4])"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "48c556f2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 4, 50257])"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4170eef5",
   "metadata": {},
   "source": [
    "- 我们将在下一章训练该模型；\n",
    "- 关于模型规模的补充说明：此前我们称其为1.24亿参数模型，可通过如下方式复核参数总量：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "f5d9cef9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "163009536"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "total_paras = sum(p.numel() for p in model.parameters())\n",
    "\n",
    "total_paras"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "49070780",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([50257, 768])"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.tok_emb.weight.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "4ccd93ee",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([50257, 768])"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.out_head.weight.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8989a96",
   "metadata": {},
   "source": [
    "- 如上所示，该模型实际参数量为1.63亿而非1.24亿，原因是什么？\n",
    "- 在原始GPT-2论文中，研究人员采用了​​权重共享​​技术，即复用词元嵌入层（tok_emb）作为输出层，具体实现为设置 self.out_head.weight = self.tok_emb.weight词元嵌入层将50,257维的独热编码输入词元投影至768维嵌入表示；\n",
    "- 输出层则将768维嵌入表示反向投影至50,257维空间，以便将结果转换回词汇形式（详见后续章节）；\n",
    "- 因此，嵌入层与输出层具有相同的权重参数规模，这一点可通过其权重矩阵的形状直接验证；\n",
    "- 关于模型规模的补充说明：此前我们称其为1.24亿参数模型，可通过如下方式复核参数总量：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "6782e28f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<generator object Module.parameters at 0x7e3981c74ba0>"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.out_head.parameters()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "1ccd5e16",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "124412160"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gpt2 = total_paras - sum(p.numel() for p in model.out_head.parameters())\n",
    "gpt2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d28b231a",
   "metadata": {},
   "source": [
    "- 在原始GPT-2论文中，研究人员将​​词元嵌入矩阵​​重复用作​​输出层的权重矩阵​​；\n",
    "- 相应地，若从总参数量中​​扣除输出层的参数​​，即可得到一个​​1.24亿参数​​的模型；"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "828f25b7",
   "metadata": {},
   "source": [
    "- 实践中，发现不采用权重共享（weight-tying）更易于模型训练；\n",
    "- 但我们将在第五章加载预训练权重时应用此项权重共享技术；\n",
    "- 最后，可通过如下方式计算模型的内存占用量，该值可作为重要的基准参考：\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf7cd46c",
   "metadata": {},
   "source": [
    "    - **GPT2-small** (the 124M configuration we already implemented):\n",
    "        - \"emb_dim\" = 768\n",
    "        - \"n_layers\" = 12\n",
    "        - \"n_heads\" = 12\n",
    "\n",
    "    - **GPT2-medium:**\n",
    "        - \"emb_dim\" = 1024\n",
    "        - \"n_layers\" = 24\n",
    "        - \"n_heads\" = 16\n",
    "    \n",
    "    - **GPT2-large:**\n",
    "        - \"emb_dim\" = 1280\n",
    "        - \"n_layers\" = 36\n",
    "        - \"n_heads\" = 20\n",
    "    \n",
    "    - **GPT2-XL:**\n",
    "        - \"emb_dim\" = 1600\n",
    "        - \"n_layers\" = 48\n",
    "        - \"n_heads\" = 25"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95d3c901",
   "metadata": {},
   "source": [
    "## 文本生成\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84ea835f",
   "metadata": {},
   "source": [
    "- GPT等大语言模型，其核心机制为​​逐词生成；"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69fe1903",
   "metadata": {},
   "source": [
    "![](./images/context.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84c8d091",
   "metadata": {},
   "source": [
    "- 实现了​​贪婪解码​​，一种简单高效的文本生成方法；\n",
    "- 在贪婪解码中，模型在每一步选择​​概率最高​​的词元作为输出（最高逻辑值对应最高概率，因此技术上无需显式计算SoftMax函数），下一章我们将实现更先进的generate_text函数；\n",
    "- 下图展示了GPT模型如何根据输入上下文生成下一个词元的过程；"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "54f2b32d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_text_simple(model, idx, max_new_tokens, context_size):\n",
    "    for _ in range(max_new_tokens):\n",
    "        idx_cut = idx[:, -context_size:]\n",
    "\n",
    "        with torch.no_grad():\n",
    "            logits = model(idx_cut)\n",
    "        \n",
    "        logits = logits[:, -1, :]\n",
    "        prob = torch.softmax(logits, dim=-1)\n",
    "        idx_next = torch.argmax(prob, dim=-1, keepdim=True)\n",
    "\n",
    "        idx = torch.cat((idx, idx_next), dim=-1)\n",
    "    \n",
    "    return idx\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7909e153",
   "metadata": {},
   "source": [
    "![](./images/gen_context.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60032bed",
   "metadata": {},
   "source": [
    "![](./images/predict.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "0e780cff",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[15496, 11, 314, 716]"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "start_context = \"Hello, I am\"\n",
    "\n",
    "encode = tokenizer.encode(start_context)\n",
    "encode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "b9e28f99",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[15496,    11,   314,   716]])"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encode_tensor = torch.tensor(encode).unsqueeze(0)\n",
    "encode_tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "c00c391c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[15496,    11,   314,   716, 27018, 24086, 47843, 30961, 42348,  7267]])"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.eval()\n",
    "\n",
    "output = generate_text_simple(\n",
    "    model=model, \n",
    "    idx=encode_tensor, \n",
    "    max_new_tokens=6, \n",
    "    context_size=GPT_CONFIG_124M[\"context_length\"]\n",
    ")\n",
    "\n",
    "output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "56a40717",
   "metadata": {},
   "outputs": [],
   "source": [
    "decode_context = tokenizer.decode(output.squeeze(0).tolist())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "e38e7544",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Hello, I am Featureiman Byeswickattribute argue'"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "decode_context"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3261150b",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "build-LLM-from-scratch",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
